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COMPUTERIZED  TRAINING  IN  CRITICAL  THINKING  (CT)2: 
A  SKILL-BASED  PROGRAM  LOR  ARMY  PERSONNEL 

EXECUTIVE  SUMMARY 


Research  Requirement: 

This  report  describes  the  development  and  assessment  of  distance  training  in  critical  thinking 
(CT)  for  Army  personnel.  The  United  States  military  has  served  a  leadership  role  in  promoting 
thinking  skills  for  good  reason.  Today’s  military  situations  demand  novel,  yet  insightful, 
solutions,  which  can  only  be  derived  from  sharply  honed  thinking  skills.  Traditional  methods  of 
training  tactical  decision-making  offer  a  prescriptive  model  that  corresponds  to  doctrine  and 
focuses  on  the  products  of  decision-making.  The  procedural  nature  of  the  doctrinal  methods  may 
actually  discourage  the  application  or  development  of  thinking  skills,  inhibiting  the  creation  of 
novel  solutions  that  might  be  the  result  of  CT  (Fallesen,  Michel,  Lussier,  &  Pounds;  1996). 

Procedure: 

The  primary  product  of  this  research  program  was  easily  distributed  and  accessed  training  in 
CT  skills,  Computerized  Training  in  Critical  Thinking  (CT)2.  The  development  of  the  training 
was  supported  by  research  that  was  conducted  over  three  broad  phases.  An  overview  of  this 
research  program  is  given  in  Fischer,  Spiker  and  Riedel  (2008a).  The  third  phase  of  the  research, 
which  is  the  focus  of  the  present  report,  was  devoted  to  developing  and  evaluating  a  complete 
Internet-based  training  package  that  provides  training  and  assessment  for  eight  CT  skills.  Two 
versions  of  the  training  were  developed:  a  standard  version  for  each  of  the  eight  CT  skills,  and  a 
longer  version  for  each  of  four  skills.  A  user-centered  design  process  was  used  to  ensure  that  the 
resulting  training  and  assessment  products  were  usable,  useful,  and  well  accepted  by  potential 
training  populations.  Consistent  with  the  user-centered  design  philosophy,  four  formative 
evaluations  were  conducted.  Feedback  obtained  from  the  formative  evaluations  was  used  to 
inform  and  make  modifications  to  the  training  system.  The  effectiveness  of  the  resulting  training 
system  was  evaluated  in  two  additional  evaluations.  Samples  of  potential  users  were  asked  to 
complete  portions  of  the  training.  Their  abilities  were  tested  before  and  after  the  training  to 
determine  if  the  training  had  improved  their  critical  thinking  skills.  One  of  the  evaluations  also 
assessed  whether  the  extended  version  of  the  training  provided  any  greater  learning  benefit  than 
the  shorter  standard  version. 

Findings: 

(CT)  is  a  modular  training  system  that  is  accessed  from  the  Internet  and  can  be  self- 
administered.  It  comprises  a  number  of  training  and  assessment  components  including  pretests, 
training  modules,  and  posttests  for  each  of  eight  CT  skills.  The  pretests  and  training  modules  are 
highly  interactive,  include  multiple  exercises  with  corresponding  feedback,  and  utilize  multi- 
media  presentations.  All  of  the  components  are  designed  not  only  to  improve  critical  thinking 
skills,  but  also  to  increase  self-awareness  of  one’s  thinking.  (CT)2  is  based  on  a  model  of  CT 
which  was  developed  during  the  research  effort  and  which  has  received  initial  empirical  support 


v 


2 

(Fisher,  Spiker  &  Riedel;  2008b).  (CT)"  was  designed  using  well-founded  pedagogical 
principles.  The  particular  CT  skills  on  which  the  training  system  focuses  were  empirically 
identified  in  the  first  phase  of  the  research  as  important  and  problematic  to  Anny  battle 
command.  The  (CT)2  software  itself  is  SCORM  compliant  and  ready  for  implementation  with 
any  learning  management  system  (LMS). 

The  formative  evaluations  conducted  during  (CT)  development  found  that  the  training  was 
generally  well  designed  although  many  specific  improvements  were  needed.  The  results  of  both 
summative  evaluations  indicated  that  the  training  increased  participants’  skills  on  the  two  critical 
thinking  skills  evaluated.  These  findings  suggest  that  the  training  program  is  effective  at 
improving  critical  thinking.  Because  the  training  and  testing  components  of  each  of  the  eight 
skills  follow  a  similar  pedagogy  and  presentation  fonnat,  it  is  expected  that  each  training  module 
would  produce  a  similar  learning  effect.  The  results  of  the  second  summative  evaluation  suggest 
that  the  shorter  standard  training  is  as  effective  as  the  extended  training  in  teaching  critical 
thinking  skills.  Given  the  limited  time  often  available  for  extra  training,  these  results  suggest  that 
the  standard  version  is  sufficient  to  produce  a  desired  learning  effect.  The  extended  version  could 
be  used  if  a  student  wanted  more  explanation  or  needed  some  question  resolved.  The  extended 
version  could  serve  as  supplemental  material,  which  could  be  made  available  for  those  trainees 
who  desire  additional  training. 

Utilization  and  Dissemination  of  Findings: 

(CT)"  is  available  for  use  by  any  Anny  organization.  Army  units  seeking  to  improve  the 
critical  thinking  skills  that  are  important  and  problematic  in  battle  command  will  find  it 
particularly  useful.  It  is  accessible  on  the  Internet  and  is  suitable  for  use  as  stand-alone  self 
development  training.  It  can  also  be  inserted  into  a  school  house  course  in  critical  thinking,  either 
using  selected  modules  or  in  its  entirety. 

The  results  of  this  multi-year  research  program  bring  two  important  resources  to  the  field  of 
critical  thinking  that  were  previously  unavailable  to  researchers.  The  first  is  a  testable  model  of 
CT  that  can  guide  future  research  efforts  on  the  construct.  The  second  is  (CT) 2,  which  provides 
an  off-the-shelf  training  package  and  a  model  on  which  future  training  development  efforts  can 
be  launched  and  which  provides  a  vehicle  for  research  on  critical  thinking. 
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COMPUTERIZED  TRAINING  IN  CRITICAL  THINKING  (CT)2: 

A  SKILL-BASED  PROGRAM  LOR  ARMY  PERSONNEL 

INTRODUCTION 

The  United  States  military  has  served  a  leadership  role  in  promoting  thinking  skills  for  good 
reason.  Military  leaders  have  always  had  to  make  tactical  decisions  in  complex  and  stressful 
situations  where  knowledge  is  incomplete  and  uncertain,  and  when  sound  thinking  can  make  the 
difference  between  success  and  failure.  But  today’s  asymmetrical  warfare  is  very  unlike  the 
warfare  that  Army  personnel  have  previously  experienced  or  studied,  and  therefore  requires 
innovative  thinking.  Today’s  situations  demand  novel,  yet  insightful,  solutions,  which  can  only 
be  derived  from  sharply  honed  thinking  skills. 

Officers  and  enlisted  personnel  are  thoroughly  educated  in  the  practice  and  art  of  warfare. 
Traditional  methods  of  training  tactical  decision-making  offer  a  prescriptive  model  that 
corresponds  to  doctrine  and  focuses  on  the  products  of  decision-making.  The  application  of 
doctrinal  procedures  may  actually  discourage  the  application  or  development  of  thinking  skills, 
inhibiting  the  creation  of  novel  solutions  that  might  be  the  result  of  CT  (Fallesen,  Michel, 

Lussier,  &  Pounds,  1996). 

Without  explicit  training,  whatever  thinking  skills  a  Soldier  possesses  are  gained  through 
on-the-job  experience,  fortuitous  experiences  in  training  exercises,  individual  disposition,  or 
other  idiosyncratic  means  such  as  self-study.  Establishing  an  integrated  training  program  to 
address  the  development  of  thinking  skills  is  preferable  to  hoping  that  these  skills  will  somehow 
develop  on  their  own.  Therefore,  the  primary  purpose  of  this  research  program  was  to  provide 
easily  distributed  and  accessed  training  in  CT  skills.  The  development  of  the  training  was 
supported  by  research  that  was  conducted  over  three  broad  phases.  In  the  next  section,  these 
three  phases  will  be  briefly  reviewed  in  order  to  provide  a  context  to  the  present  report. 

This  report  describes  the  main  product  of  a  multi-year  research  and  development  program  to 
create  distance  training  in  critical  thinking  for  Anny  personnel.  The  training  that  was  developed, 
Computerized  Training  in  Critical  Thinking  (CT)2  is  a  modular,  interactive  training  system  that 
is  accessed  from  the  Internet  and  can  be  self-administered.  It  comprises  a  number  of  training  and 
assessment  components  including  pretests,  training  modules,  and  posttests  for  each  of  eight  CT 
skills.  Four  of  the  CT  skills  have  an  extended  training  version  in  addition  to  the  standard  shorter 
version.  The  pretests  and  training  modules  are  interactive,  include  multiple  exercises  with 
feedback,  and  utilize  multi-media  presentations.  All  of  the  components  are  designed  to  not  only 
improve  critical  thinking  skills,  but  also  to  increase  self-awareness  of  one’s  thinking.  (CT)2  is 
based  on  a  model  of  CT,  developed  during  this  research  effort,  which  has  undergone  initial 
empirical  testing  (Fischer,  Spiker  &  Riedel;  2008b).  (CT)2  was  developed  using  well-founded 
pedagogical  principles.  The  particular  CT  skills  on  which  the  system  focuses  were  empirically 
identified  in  the  first  phase  of  the  research  as  important  in  the  battle  command  domain  and  whose 
execution  was  problematic  (Fischer,  Spiker,  &  Riedel,  2008c).  Here  battle  command  applies  to 
“the  leadership  element  of  combat  power...  Commanders  visualize  the  operation,  describe  it  in 
tenns  of  intent  and  guidance,  and  direct  the  actions  of  subordinates  within  their  intent.  They 


1 


directly  influence  operations  by  personal  presence...”  (Department  of  the  Army,  2001,  Section 
5-1).  ' 


The  next  section  of  this  report  gives  a  brief  overview  of  the  multi-year  research  program  in 
critical  thinking  that  provides  the  foundation  for  (CT)“.  Succeeding  sections  describe  the 
curriculum,  the  training  components,  and  the  formative  and  summative  evaluations  of  (CT)2. 
Finally,  suggestions  for  future  research  are  presented. 
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OVERVIEW  OF  THE  RESEARCH  PROGRAM 


2 

In  the  first  phase  in  the  development  of  (CT)-,  which  constituted  preparatory  research,  the 
CT  literature  was  reviewed,  CT  was  investigated  in  a  military  context,  and  pedagogical 
techniques  used  to  teach  CT  were  surveyed.  In  the  second  phase,  a  prototype  CT  training 
program  was  developed  for  two  CT  skills.  The  final  phase,  which  is  the  focus  of  the  present 
report,  was  devoted  to  developing  a  complete  Internet-based  training  package  that  provides 
training  and  assessment  for  eight  CT  skills.  The  preparatory  research  and  development  of  the 
prototype  (CT)2  are  discussed  below. 


Preparatory  Research 

The  preparatory  phase  conducted  the  necessary  preliminary  research  that  would  support 
development  of  (CT)2.  The  major  tasks  in  this  first  phase  were  to  (1)  review  the  literature  on  CT, 
(2)  develop  and  validate  a  model  of  CT,  (3)  inventory  CT  skills  identified  in  the  literature,  (4) 
specify  the  role  of  CT  within  the  Army,  (5)  select  a  set  of  CT  skills  for  which  to  develop 
training,  (6)  establish  lessons  learned  from  the  literature  on  training  CT  and  from  efforts  to  teach 
CT  to  Army  officers,  and  (7)  establish  functional  requirements  for  the  to-be-developed 
instructional  material.  A  more  detailed  description  of  this  research  is  contained  in  Fischer, 
Spiker,  and  Riedel  (2008b). 

The  review  of  the  literature  on  CT  concluded  that  the  field  is  highly  fragmented  and  lacks 
consensus  (Fischer,  et  ah,  2008b).  A  lack  of  empirical  investigation  of  CT  as  a  psychological 
process  and/or  individual  difference  variable  is  a  major  problem  with  this  field  and  may  be  one 
reason  for  its  lack  of  coherence.  The  review  showed  that  the  literature  on  CT  did  not  contain  a 
model  of  CT  that  grounded  the  construct  in  objectively  observable  behavior.  The  literature 
review  also  produced  an  inventory  of  CT  skills  and  provided  input  to  development  of  a  testable 
model  of  CT  (Fischer,  et  al.,  2008b),  which  was  one  of  the  most  important  products  of  the 
preparatory  research. 


An  empirically  testable  model  was  developed  and  two  studies  were  conducted  to  provide  an 
initial  validation  of  the  model.  The  model  incorporates  many  ideas  about  CT  offered  by  leading 
thinkers  in  philosophy  and  education  and  embodies  many  of  the  CT  skills  and  predisposing 
attitudes  discussed  in  the  CT  literature.  It  also  specifies  the  relationships  among  a  variety  of 
variables  including  the  influence  of  experience  and  knowledge,  and  the  relationship  of  CT  to 
cognitive  tasks  (e.g.,  judgment  and  problem  solving).  The  model  also  provides  a  framework  in 
which  CT  can  be  empirically  investigated  as  a  cognitive  process.  Figure  1  shows  the  model.  A 
description  and  discussion  of  the  model  is  contained  in  Appendix  A. 

A  validation  experiment  tested  four  of  the  model’s  predictions  involving  the  effects  of 
situation  (type  of  material  to  be  processed),  type  of  task,  predisposing  individual  differences,  and 
experience  level  on  CT  (Fischer,  et  al.,  2008b).  It  also  examined  the  model’s  proposition  that  CT 
can  have  negative  affective  consequences.  The  model’s  predictions  concerning  the  type  of 
material  to  be  processed  were  generally  supported.  High  substance  material  took  longer  and  was 
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more  effortful  to  process  than  low  substance  material.  Response  times  were  longer  for  degraded 
substance  material  than  non-degraded  substance  material. 


Initial  Input 


Processing 

Engines 


Figure  1.  Model  of  critical  thinking  (Fischer,  et  al.,  2008b). 

The  results  of  this  investigation  failed  to  support  the  hypothesis  that  predisposing  individual 
difference  factors  affect  the  tendency  to  engage  in  CT  skills.  A  “Need  for  Cognition”  (NFC) 
scale  failed  to  correlate  with  any  measure  of  CT.  However,  potential  restriction  of  range 
problems  on  the  NFC  scale  may  have  produced  this  null  effect. 

The  investigation  further  found  that  experience  level  does  affect  the  application  of  CT  skills. 
The  investigation  also  found  that  negative  affect  was  not  a  simple,  direct  outcome  of  CT  in  this 
investigation.  Instead,  it  appears  that  other  factors  such  as  type  of  task  and  type  of  material 
detennined  the  type  of  affect  experienced  by  participants. 

In  summary,  the  model  of  CT  generated  a  number  of  predictions  that  had  not  been 
previously  empirically  tested.  The  model  was  sufficiently  specified  to  pennit  falsification  of 
many  of  its  assertions.  The  validation  experiment  tested  five  of  its  central  predictions,  three  of 
which  were  supported. 

The  preparatory  research  also  examined  the  role  of  CT  within  the  Army  and  identified  either 
high  impact  critical  thinking  skills  to  train.  (Fischer,  Spiker,  &  Riedel,  2008b).  Eighteen  Army 
officers  at  Ft.  Hood,  Texas,  completed  a  survey  that  assessed  their  opinions  and  experiences 
concerning  CT  skills,  and  predisposing  attitudes  and  situational  conditions  within  the  battle 
command  related  to  CT.  The  officers  were  asked  to  rate  how  important  each  of  thirteen  broad 
categories  of  CT  skills  were  to  battle  command,  where  battle  command  applies  “to  the  leadership 
element  of  combat  power... Commanders  visualize  the  operation,  describe  it  in  terms  of  intent 
and  guidance,  and  direct  the  actions  of  subordinates  within  their  intent.  They  directly  influence 
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operations  by  personal  presence,  supported  by  their  command  and  control  (C2)  system.” 
(Department  of  the  Army,  2001,  p.  89).  They  were  also  asked  to  indicate  if,  in  their  experience, 
they  had  observed  problems  in  the  execution  of  each  broad  skill  category.  The  survey  results 
were  then  used  to  select  several  categories  of  skills  that  are  key  to  effective  battle  command. 
Selecting  a  subset  of  key  CT  skills  from  the  set  of  13  skill  categories  evaluated  by  the  officers 
was  accomplished  by  applying  two  criteria  to  the  Fort  Hood  survey  data:  the  mean  importance 
rating  and  the  reported  incidence  of  problems  in  executing  the  CT  skills.  By  jointly  considering 
the  two  criteria  in  the  selection  process,  eight  CT  skills  were  selected  and  became  the  target 
skills  to  be  trained  in  (CT)2.  The  results  also  indicated  that  the  CT  model  largely  captures  the 
skills,  situational  conditions,  and  predisposing  factors  significant  to  Army  leadership. 

Before  developing  the  prototype  for  (CT)“  however,  lessons  learned  were  gathered  from  (1) 
the  civilian  research  literature  on  training  CT  and  (2)  current  efforts  to  teach  CT  to  Army 
officers.  The  literature  review  (Fischer,  et  al.,  2008b)  found  that  educators  have  used  a  wide 
variety  of  methods  to  teach  CT.  These  were  inventoried  in  Fischer,  et  al.  ,2008b.  However,  only  a 
small  proportion  of  those  methods  have  been  empirically  validated.  Lessons  learned  from 
educators  at  the  Command  General  Staff  College  (CGSC)  at  Fort  Leavenworth,  Kansas  State 
University,  and  the  Army  War  College  revealed  a  number  of  issues  (Fischer,  et  al.,  2008b). 
Respondents  reported  that  the  biggest  difficulty  instructors  experienced  came  from  the  Army 
itself.  Specifically,  Anny  culture  tends  to  discourage  thinking  to  some  extent.  Because  of  the 
Army’s  hierarchical  organization  and  the  requirement  for  unit  cohesiveness,  thinking  “out  of  the 
box”  is  often  discouraged  by  leaders  and  by  doctrine.  For  this  reason,  courses  in  CT  that  are 
electives  typically  have  low  enrollments,  and  students  sometimes  balk  at  the  content  in  CT 
courses. 

A  final  effort  of  the  preparatory  research  was  to  integrate  the  infonnation  that  had  been 
gathered  to  establish  functional  requirements  for  (CT)2.  The  research  concluded  that  a  CT 
training  system  would  need  to  be  (1)  practically  integrated  with  current  Army  training  methods, 
(2)  administered  at  multiple  levels  of  training,  and  (3)  readily  accessible  by  instructors  and 
students  in  a  variety  of  settings  such  as  the  schoolhouse  or  field  unit.  With  these  functional 
requirements  in  mind,  the  second  phase  of  the  research  program  was  initiated.  A  complete 
discussion  of  the  research  and  results  conducted  in  the  preparatory  phase  can  be  found  in  Fischer 
Spiker,  and  Riedel  (2008a,  2008b,  2008c). 

Prototype  Development  and  Evaluation 
Prototype  Training  System  for  Two  Skills 

The  CT  model  provided  the  basis  on  which  to  design  and  develop  a  prototype  training 
system.  The  prototype  system  targeted  two  of  the  eight  CT  skills  previously  identified.  The  two 
skills  were  “Frame  the  Message”  and  “Recognize  Gist  in  Material”.  Because  the  model  proposes 
that  CT  is  a  deliberate,  systematic  awareness  of  the  process  and  products  of  one’s  own  thinking, 
the  prototype  training  focused  on  increasing  self-awareness.  It  also  targeted  common,  and 
potentially  serious,  errors  that  people  make  when  they  fail  to  apply  the  two  CT  skills.  The 
prototype  training  program  highlighted  awareness  of  common  errors  and  taught  specific 
techniques  that  can  be  used  to  overcome  them.  It  presented  the  student  with  real-world  situations 
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and  asked  them  to  complete  numerous  thinking  exercises  that  require  the  practice  and  application 
of  CT  skills  in  a  variety  of  realistic  settings. 

Evaluation  of  Prototype  Training 

To  evaluate  the  effectiveness,  usability,  and  student  acceptance  of  the  prototype  training 
system,  an  empirical  evaluation  was  conducted  during  the  second  phase  of  the  research  program 
(Fischer,  Spiker,  &  Riedel,  2008c).  The  central  objective  of  the  evaluation  was  to  determine 
whether  the  prototype  CT  training  system  effectively  increases  measurable  indicators  of  CT 
compared  to  two  other  learning  conditions.  The  investigation  also  assessed  participating 
students’  attitudes  and  subjective  evaluations  of  the  training  as  indicators  of  acceptance  and 
usability.  The  effectiveness  of  the  prototype  system  was  evaluated  in  an  investigation  in  which 
participants  from  the  85th  Reserve  Training  Division  worked  through  parts  of  the  training.  The 
results  of  this  investigation  indicated  that  military  students  found  the  training  highly  acceptable. 
Although  the  sample  of  participants  who  used  the  training  was  small,  it  was  uniformly  positively 
rated.  Users  found  it  interesting  and  well  worth  their  time.  The  prototype  training  also  appeared 
to  be  generally  effective  at  encouraging  critical  thinking,  at  least  about  messages  Army  personnel 
must  evaluate.  The  investigation  showed  that  the  web-based  prototype  enhanced  memory  for 
messages,  possibly  because  it  encourages  greater  depth  of  processing. 

The  evaluation  also  showed  that  the  training  inhibited  the  production  of  (potentially 
incorrect)  inferences  that  go  well  beyond  what  is  explicitly  given  in  the  message.  Participants 
who  took  the  training  made  far  fewer  unjustified  inferences  than  participants  assigned  to  the 
other  two  training  conditions.  Examination  of  the  responses  reveals  that  training  participants  did 
make  inferences;  however,  they  justified  them  by  pointing  out  explicit  infonnation  given  in  the 
message  that  supported  their  inferences.  Therefore,  the  prototype  system  appeared  to  encourage 
discrimination  of  what  is  “known”  or  “given”  from  what  might  be  added  (i.e.,  inferred)  by  the 
perceiver. 


Development  and  Evaluation  of  (CT)2 

The  final  phase  of  the  research  program,  which  is  the  focus  of  the  present  report,  was 
devoted  to  developing  a  complete  Internet-based  training  package  that  provides  training  and 
assessment  components  for  eight  CT  skills.  The  training  component  contains  an  extended 
training  version  for  four  of  the  eight  skills. 

A  user-centered  design  process  was  used  to  ensure  that  the  resulting  training  and  assessment 
products  were  usable,  useful,  and  well  accepted  by  potential  training  populations.  Early  in  the 
development  of  (CT)2,  a  panel  of  Army  training  experts  was  convened  to  review  early 
conceptions  of  its  design.  A  systematic  design  process  was  also  used,  starting  with  the 
development  of  functional  specifications  for  each  training  component,  proceeding  to 
storyboarding  and  content  development,  and  then  moving  to  implementing  the  content  on  the 
web.  Consistent  with  the  user-centered  design  philosophy,  four  formative  evaluations  were  also 
conducted.  In  each  investigation,  Army  Soldiers  or  reservists  worked  through  and  evaluated  one 
or  more  of  the  training  or  assessment  components.  The  resulting  curriculum  is  described  in  the 
next  section. 
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OVERVIEW  OF  (CT)2  CURRICULUM 


We  begin  with  a  brief  description  of  the  purpose  of  the  training  as  well  as  the  student 
population  to  which  this  training  is  targeted.  We  then  outline  the  specific  critical  thinking  skills 
that  the  training  addresses.  The  next  section  provides  a  detailed  account  of  the  five-step 
development  process  that  was  used  and  includes  technical  specifications,  content  review, 
storyboarding,  software  development,  and  formative  evaluations.  The  section  concludes  with  a 
description  of  the  products  that  comprise  the  training,  including  pretests,  posttests,  and  the 
training  modules  themselves. 

Purpose  of  Critical  Thinking  Training 

The  overarching  purpose  of  (CT)  is  to  provide  students  and  their  instructors  with  a 
framework  for  acquiring  knowledge  of  eight  key  critical  thinking  skills,  learning  the  basic 
concepts  underlying  their  application,  and  receiving  interactive  practice  in  exercising  these  skills 
in  the  context  of  realistic  decision-making  scenarios  using  actual  battle  command  materials. 
(CT)2  is  intended  to  raise  students’  awareness  of  the  need  for  CT  and  of  their  own  use  of  CT. 
Pretests  were  developed  for  each  skill  and  posttests  were  provided  as  a  means  to  gauge  that 
students  had  achieved  the  enabling  and  terminal  objectives  upon  exiting  the  training  module. 

Intended  Population 

(CT)'  was  originally  designed  for  Army  captains  and  majors  (0-3  and  0-4).  However,  as 
content  development  proceeded,  and  iterative  usability  testing  with  uniformed  personnel  was 
performed,  it  became  clear  that  many  of  the  foundational  principles  of  critical  thinking  would  be 
equally  applicable  to  Anny  personnel  of  lower  ranks,  including  Lieutenants  and  Non- 
Commissioned  Officers  (NCOs).  Thus  (CT)2  was  designed  for  a  broad  target  audience,  with  few 
prerequisites  needed.  Scenarios  and  vignettes  were  developed  that  require  only  modest  exposure 
to  the  principles  of  command  and  control,  the  military  decision  making  process  (MDMP),  basic 
Army  symbology,  operations  orders,  and  reporting  formats. 

This  broader  target  population  now  includes  advanced  undergraduates  in  the  ROTC 
program,  West  Point  cadets,  National  Guard  and  Reservists,  as  well  as  the  more  senior  captains, 
majors,  and  even  lieutenant  colonels  and  colonels  (0-5,  0-6).  The  original  intent  was  to  restrict 
our  training  to  officers,  but  usability  reviews  and  interviews  with  recently  returning  Operation 
Iraqi  Freedom  (OIF)  veterans  (see  formative  evaluations  later  in  this  report)  showed  that  NCOs 
are  also  an  appropriate  group.  In  today’s  military,  NCO’s  have  important  command 
responsibilities,  and  must  utilize  critical  thinking  skills  on  a  daily  basis,  just  like  their  officer 
counterparts.  Hence,  the  overall  target  audience  for  (CT)2  is  now  any  Army  personnel  who  will 
be  required  to  review,  comprehend,  and  respond  to  battle  command  materials.  At  the  very  least, 
that  would  be  any  Soldier  who  would  be  assigned  to  a  tactical  operations  center  (TOC)  at  the 
battalion,  brigade,  division,  or  corps  echelon. 
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Critical  Thinking  Skills  Trained  in  (CT)2 

The  (CT)  approach  to  training  critical  thinking  is  both  data-driven  and  model-based. 
Empirical  data  were  collected  to  determine  which  critical  thinking  skills  were  most  important 
and  most  in  need  of  training.  This  entailed  conducting  in-depth  interviews  with  19  Anny  officers 
at  Fort  Hood  (Fischer,  et  ah,  2008b).  The  results  of  this  survey  revealed  the  types  of  errors  that 
are  most  often  made  during  battle  command  tasks,  and  thus  guided  the  content  of  our  training. 
The  model  of  critical  thinking,  described  in  Appendix  A,  was  then  used  to  detennine  how  to  train 
the  most  important  of  these  high-payoff  skills.  Because  the  model  of  critical  thinking — and  its 
emphasis  upon  two  systems  of  information  processing — is  unique,  the  approach  to  training 
critical  thinking  is  unique  as  well.  As  discussed  in  later  sections,  we  emphasize  thinking  by 
doing,  where  the  learner  is  led  to  make  typical  thinking  errors.  Critical  thinking  skills  are  then 
taught  to  help  identify  those  errors,  correct  them,  and  thus  avoid  them  in  the  future. 

(CT)“  is  not  intended  to  promote  perfect,  error-free  perfonnance.  Rather,  the  techniques  are 
intended  to  help  Army  personnel  catch  thinking  errors  that  do  occur,  and  correct  them,  before 
performance  degrades.  The  ability  to  recognize  one’s  own  thinking  errors  is  the  hallmark  of  a 
good  critical  thinker. 

The  eight  high-payoff  CT  skills  trained  in  (CT)2  are  summarized  in  Table  1.  The  left-hand 
column  gives  the  name  of  the  skill,  while  the  second  column  offers  a  working  definition  of  the 
skill  in  practical  tenns.  The  third  column  describes  the  battle  command  tasks  most  directly 
supported  by  the  skill,  with  the  right-most  column  specifying  the  errors  and  problems  most 
frequently  reported  with  that  task.  The  skills  are  listed  in  no  particular  order.  The  (CT)2  User’s 
Manual  (Fischer,  Spiker,  Harris,  &  McPeters,  2004),  describes  how  the  trainer  might  wish  to 
organize  the  sequence  of  skills  in  order  to  maximize  training  effectiveness. 


Table  1.  Overview  of  Eight  High-Payoff  Critical  Thinking  Skills 


Skill 

Definition 

Primary  Battle 
Command  Task 

Battle  Command  Errors  and 

Deficiencies 

1.  Frame  the  Message 

The  ability  to  identify  the  essential 
elements  of  a  message,  understand 
their  relationships,  and  describe  a  high 
fidelity  representation  of  the  message. 

Clarify  the  intent  of  the 
commanders  1  and  2 
levels  up 

Difficulty  in  establishing  clear  and  accurate 
understanding  of  commander’s  intent 

Difficulty  in  conveying  clear  commander’s  intent 

2.  Recognize  Gist  in 
Material 

The  ability  to  sort  through  the  details 
in  a  message  (written,  graphical, 
visual,  auditory,  and/or  tabular)  and 
extract  the  gist  therein. 

Restate  mission 
objectives  provided  by 
upper  echelon  to  write 
own  mission  statement 

Too  much  detail  in  operations  orders  (OPORDs) 
that  must  be  filtered  to  establish  gist  that  supports 
writing  of  own  mission  statement 

Too  little  time  at  lower  echelons  to  accurately 
extract  essence  of  mission 

3  Develop  an 
Explanation  that 

Ties  Information 
Elements  Together 
in  a  Plausible  Way 

The  ability  to: 

•  Arrange  evidence  logically 

•  Highlight  the  gaps  in  knowledge. 

•  Develop  an  explanation  or  multiple 
explanations  based  on  evidence 

•  Evaluate  explanation(s)  for 
plausibility 

Interpret  reports  of 
recent  enemy  activities 
in  area  of  interest  to 
estimate  enemy  intent 
and  predict  enemy 
actions 

Overlook  seemingly  unrelated  facts 

Fail  to  assess  the  quality  of  information 

Difficulty  in  filtering  excessive  information 
Tendency  to  embellish  enemy  activity  reports — 
over-reports  of  enemy  contact  and  movement 
Tendency  to  discount  initial  reports 

4.  Generalize  from 
Specific  Instances  to 
Broader  Classes 

The  ability  to  recognize  and  then 
classify  specific  facts/incidents/events 
as  part  of  a  general  category. 

Interpret  reports  of 
enemy  disposition 

Fail  to  accurately  induce  patterns  of  overall 
movement  based  on  report  instances 

Tendency  to  disregard  reports  that  do  not  match 
expectations 

Tendency  to  inflate  information  in  reports 

5.  Use  Mental  Imagery 

The  ability  to  accurately  create  mental 
images  in  one’s  mind  of  how 
resources  will  be  applied  and  events 
will  unfold  within  a  situation. 

Develop  scheme  of 
maneuver 

War  game  courses  of 
action  (COAs) 

Fail  to  visualize  events 

Fail  to  include  sufficient  detail  in  COAs 

Fail  to  consider  contingencies 

Fail  to  consider  how  plans  could  go  wrong 

Generate  only  one  COA 

Fail  to  consider  combat  multipliers 

Difficulty  in  keeping  track  of  mobile  forces 

6.  Challenge  One’s 

Bias 

The  ability  to  consistently  reevaluate 
one’s  current  view  of  the  situation  for 
prejudice  or  bias  as  new  information 
is  received. 

Change  own-unit  plans 
based  on  new  tactical 
input 

Tendency  to  “fight  the  plan” 

General  reluctance  to  change  plans 

7.  Examine  Other 
Peoples’ 

Perspectives 

The  ability  to  view  and  interpret  a  set 
of  circumstances  from  the  perspec¬ 
tives  of  different  individuals,  different 
cultures/religions,  and  different 
timeframes  (historical  perspective). 

Interpret  reports  of 
recent  enemy  activities 
in  area  of  interest 

Failure  to  accurately  estimate  enemy  intent 

8.  Decide  When  to 

Seek  Information 
Based  on  its  Value 
and  Cost 

The  ability  to  evaluate  the  need  for 
new  information  in  terms  of  its  cost 
in:  time,  resources,  risk 

Assess  current  situation 

Tendency  to  spend  too  much  time  planning  and 
gathering  information 

Tendency  to  make  quick  decisions  without 
gathering  more  information 
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DESCRIPTION  OF  TRAINING 
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(CT)“  is  modularized  into  several  components  for  the  eight  separate,  but  complementary, 
skills.  The  training  modules  and  associated  tests  can  be  selected  individually  from  a  learning 
management  system.  Figure  2  shows  that  a  pretest,  training  module  and  posttest  are  available  for 
each  of  the  eight  skills.  An  extended  training  module  is  available  for  Skills  1,  2,  3  and  4.  These 
extended  modules  provide  more  exercises  and  more  detailed  explanations.  They  each  take  about 
12  hours  to  complete  as  opposed  to  two  hours  each  for  the  standard  modules. 


Introduction  to  Critical  Thinking. 

Critical  Thinkina  Use  Manual. 

Skill  One:  Frame  the  Message 

Skill  Two:  Recognize  Gist  in  Material 

Pretest 

Pretest 

Standard  Training  Module 

Standard  Training  Module 

Posttest 

Posttest 

Extended  Training  Module 

Extended  Training  Module 

Introduction 

Introduction 

Element  1 

Element  1 

Element  2 

Element  2 

Element  3 

Element  4 

Element  5 

Skill  Three:  Develop  an  Explanation 

Skill  Four:  Generalize  from  Specific  Instances 

Pretest 

Pretest 

Standard  Training  Module 

Standard  Training  Module 

Posttest 

Posttest 

Extended  Training  Module 
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Figure  2.  Course  modules  of  (CT)2. 
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Pretests 


A  pretest  is  provided  for  each  skill  for  the  purposes  of  1)  enabling  student  self-assessment  of 
proficiency  in  application  of  the  skill,  2)  increasing  student  sensitivity  to  the  errors  and 
deficiencies  associated  with  lack  of  or  inappropriate  skill  application,  3)  increasing  student  self 
awareness  of  CT  capability,  and  4)  presenting  feedback  to  initiate  the  process  of  learning  skill 
application.  While  the  pretest  for  each  skill  requires  about  the  same  amount  of  time  (20  minutes 
or  so),  pretests  differ  somewhat  in  the  format  employed  to  meet  pretest  objectives.  Brief 
descriptions  of  the  pretests  for  Skill  2  ( Frame  the  Message)  and  Skill  4  ( Generalize  from  Specific 
Instances  to  Broader  Classes)  are  provided  here  to  illustrate  the  types  of  formats  employed  for 
the  pretests. 

The  Skill  2  pretest  is  organized  around  three  short  passages  that  represent  the  kinds  of 
messages  typically  encountered  by  an  Army  officer,  such  as  intent  statements  or  operational 
plans.  Following  each  passage,  students  are  required  to  make  four  types  of  responses  designed  to 
assess  their  ability  to  recognize  the  gist  of  a  message:  1)  writing  a  one  sentence  summary  to 
capture  the  gist  of  the  passage,  2)  selecting,  from  a  list,  the  points  that  should  b  e  included  to 
capture  the  gist,  3)  selecting  from  a  list  the  statement  that  best  captures  the  gist,  and  4)  ranking  a 
set  of  statements  on  how  closely  each  is  related  to  the  gist  of  the  passage.  After  the  test  is 
completed,  feedback  is  provided  on  the  student’s  ability  to  capture  the  gist  of  a  message. 

The  Skill  4  pretest  is  organized  around  three  scenes  or  scenarios,  each  involving  some  type 
of  stability  and  support  operation.  Two  scenarios  are  described  via  photographs  in  which  features 
and  situations  must  be  viewed  for  a  limited  time  by  the  student  and  then  classified  by  responding 
to  questions.  The  third  scenario,  described  in  a  situation  report,  is  classified  by  answering  a  set  of 
questions.  In  each  case,  after  the  scenes  or  scenarios  are  removed  from  sight,  the  student  is  asked 
to  1)  make  a  classification  based  on  memory  of  the  defining  features  that  were  present,  2)  report 
whether  certain  critical  features  or  elements  were  present,  and  3)  identity  the  critical  features  that 
might  alter  the  classification  if  they  were  known  by  the  student  to  be  present  or  absent.  After  the 
test  is  completed,  feedback  is  provided,  in  the  form  of  a  test  score,  on  the  student’s  ability  to 
generalize  from  specific  instances  to  broader  classes. 

Posttests 

A  posttest  for  each  training  module  is  provided  for  use  in  assessing  student  comprehension 
of  the  critical  thinking  concepts  presented  in  the  module  and  for  evaluating  student  capability  for 
applying  the  skills  learned.  Posttests  vary  in  format  from  module  to  module,  depending  on  what 
testing  techniques  were  considered  most  effective  in  meeting  the  assessment  objectives  for  the 
module.  Training  administrators  may  employ  posttests  to  evaluate  student  progress  and  to 
identify  training  weaknesses.  Students  may  employ  posttests  for  self-evaluation  of  their  progress 
and  to  identity  weaknesses  in  their  critical  thinking  knowledge  and  skills. 

The  eight  posttests  employ  a  variety  of  testing  techniques.  Multiple-choice  questions  are 
used  extensively  for  testing  knowledge  acquisition  and,  combined  with  other  techniques,  for 
testing  skill  application.  Many  posttests  require  the  application  of  critical  thinking  skills  to 
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operational  scenarios  and  sample  materials  such  as  messages  and  photographs.  Fonnats  for 
student  responses  include  classification,  matching,  ranking  and  rating  by  means  of  dragging-and- 
dropping  items.  Analytical  responses  are  also  made  by  selecting  items  from  lists  by  checking 
boxes,  selecting  YES  or  NO,  or  selecting  TRUE  or  FALSE. 


Standard  Training  Modules 

A  standard  training  module  is  provided  for  each  of  the  eight  critical  thinking  skills  (see 
Table  1).  The  modules  were  developed  to  meet  Army  training  needs  and  to  adhere  to  the  four 
themes  and  18  pedagogical  principles  (Fischer  et  al.,  2008c).  Each  module  requires 
approximately  two  hours  to  complete;  however,  individual  differences  among  students  have  been 
shown  to  results  in  a  sizable  distribution  of  times  around  this  two-hour  target. 

Extended  Training  Modules 

Extended  training  modules  are  provided  for  the  skills  considered  most  fundamental  to 
critical  thinking:  Skill  1  (Frame  the  Message),  Skill  2  ( Recognize  Gist  in  Material),  Skill  3 
(Develop  an  Explanation)  and  Skill  4  (Generalize  from  Specific  Instances  to  Broader  Classes). 
Each  extended  training  module  consists  of  an  introduction  and  from  two  to  five  training 
elements.  The  organization  of  each  extended  module  is  illustrated  here  with  a  description  of  the 
Skill  1  module,  which  consists  of  an  introduction  and  five  elements.  The  other  extended  training 
modules  follow  essentially  the  same  approach  and  format. 

The  introductory  tutorial  provides  answers  to  the  following  questions:  What  is  framing? 
What  is  a  frame?  How  does  framing  help  you  think?  What  are  the  benefits  of  framing  a 
message?  What  does  a  message  frame  look  like?  What  is  the  plan  for  training  this  skill? 

In  Element  1  the  student  leams  about  structured  messages  and  their  frames,  and  practices 
identifying  correct  frames  for  messages  of  different  structures  and  matching  parts  of  structured 
messages  to  parts  of  the  frame. 

Element  2  provides  instruction  and  practice  in  analyzing  unstructured  messages.  A 
categorization  system  is  introduced  and  employed  to  categorize  message  components.  Practice  is 
then  provided  in  applying  the  system  to  messages  and,  also,  in  identifying  the  relationships  that 
exist  among  message  components. 

The  objective  of  Element  3  is  to  teach  the  student  how  to  identify  weak  spots  in  a  message. 
The  student  is  instructed  in  the  six  different  ways  that  a  message  might  contain  weak  spots.  He  or 
she  is  then  given  practice  in  identifying  these  weak  spots  in  sample  messages.  The  student  is  also 
instructed  and  given  practice  in  distinguishing  between  message  components  that  are  weak 
because  they  are  unclear  and  components  that  are  weak  because  the  information  is  uncertain. 

Element  4  addresses  the  resolution  of  weak  spots  in  a  message.  The  student  is  instructed  in 
and  given  practice  in  procedures  that  are  effective  in  resolving  message  weaknesses  due  to  lack 
of  clarity  and  uncertainty.  The  student  is  also  instructed  and  given  practice  in  gauging  the 
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strength  of  inferences  that  may  be  required  for  resolving  weak  spots  in  messages.  The  final 
element,  Element  5,  provides  instruction  in  the  difference  between  evidence  and  inference,  and 
provides  practice  in  evaluating  inferences. 

Flexible  Use  and  Progression  of  Course  Elements 

The  time  required  to  work  through  all  of  the  training  modules  in  the  (CT)“  program  may 
exceed  the  training  time  available  for  a  group  of  students.  On  average,  it  will  take  approximately 
two  hours  to  work  through  each  of  the  eight  CT  skills,  for  a  total  of  16  hours.  If  a  unit  has  only  8 
hours  available  for  CT  training,  which  4  modules  should  be  selected?  If  only  4  hours  can  be 
spared  for  training,  which  2  should  be  selected? 

As  discussed  below,  we  recommend  two  possible  courses  of  action  for  selecting 
modules/skills  for  training  when  time  is  limited.  On  the  one  hand,  one  can  examine  the  types  of 
errors  that  each  training  module  is  designed  to  address,  and  compare  those  to  known  or 
conjectured  areas  of  weakness  within  one’s  operational  or  training  unit.  Alternatively,  one  can 
arrange  the  skills  in  order  of  difficulty,  and  concentrate  on  those  CT  skills  that  match  the  unit’s 
present  CT  skill  level. 

Because  the  training  materials  for  each  skill  were  developed  as  standalone  items,  none  of  the 
training  modules  are  prerequisites  for  any  other.  Consequently,  either  method  is  valid  for 
selecting  a  subset  of  modules  for  conducting  CT  training.  We  next  briefly  describe  how  each 
method  could  be  employed. 

Recommended  Use  -  Matching  CT  Skills  to  Errors 

To  use  this  CT  skill  selection  strategy,  the  analyst  or  trainer  would  first  construct  a  list  of  the 
most  troublesome  or  vexing  errors/breakdowns  in  the  battle  command  domain  observed  in 
his/her  unit.  Then,  depending  on  the  time  available  for  training,  pick  the  CT  skills  whose  targeted 
errors  best  match  the  ones  that  have  been  occurring  in  the  field.  Since  the  listing  of  battle 
command  errors  in  Table  1  is  only  partial,  the  analyst  will  have  to  use  his/her  judgment  to  find 
the  best  matches.  In  Table  2,  we  provide  a  set  of  decision  rules  that  could  be  used  to  guide 
selection. 

Recommended  Progression  -  Matching  CT  Skills  to  Difficulty  Level 

Another  way  to  select  the  CT  skills  to  be  trained  is  based  on  estimated  difficulty  level.  Table 
3  categorizes  the  eight  CT  skills  based  on  an  estimated  level  of  challenge.  For  simplicity,  the 
skills  are  grouped  into  low,  medium,  or  high  difficulty  level.  Please  note  that  these  are  only 
estimates,  as  empirical  data  concerning  relative  skill  difficulty  have  yet  to  be  collected.  However, 
this  listing  provides  a  reasonable  starting  point  for  making  a  skill  selection  decision  when 
training  time  is  limited. 
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Table  2.  Decision  Rules  to  Guide  Selection  of  Training  Modules 


If  the  unit  is  having  problems  understanding,  translating,  or 
writing  commander’s  intent  statements  or  following 
commander  guidance 

THEN 

Use  CT  Skill  1 :  Frame  the 

Message 

If  the  unit  is  having  trouble  reading  through  complex 
OPORDs,  is  taking  too  long  in  writing  their  own  mission 
statements,  or  not  able  to  fdter  out  irrelevant  information 
from  mission  objectives 

THEN 

Use  CT  Skill  2:  Recognize  Gist  in 
Material 

If  the  unit  is  having  trouble  interpreting  reports  of  enemy 
intent,  does  a  poor  job  in  assessing  the  quality  of 
information,  or  is  inconsistent  in  reporting  enemy  contact 
and  movement 

THEN 

Use  CT  Skill  3:  Development 
Explanation  that  Ties  Information 
Together 

If  the  unit  is  having  trouble  extracting  patterns  of  movement 
based  on  individual  reports  or  is  not  able  to  inteipret  reports 
of  projected  movements  accurately 

THEN 

Use  CT  Skill  4:  Generalize  from 
Specific  Instances  to  Broader 
Classes 

If  the  unit  is  failing  to  consider  contingencies,  include 
sufficient  detail  in  COAs,  or  anticipate  how  time  and  place 
affect  how  the  world  looks 

THEN 

Use  CT  Skill  5:  Use  Mental 

Imagery 

If  the  unit  is  showing  a  general  reluctance  to  change  a  failing 
plan  (i.e.,  fighting  the  plan) 

THEN 

Use  CT  Skill  6:  Challenge  One’s 
Bias 

If  the  unit  is  failing  to  accurately  estimate  enemy  intent  or  is 
unable  to  appreciate  the  other  side’s  point  of  view 

THEN 

Use  CT  Skill  7:  Examine  other 
People’s  Perspectives 

If  the  unit  is  spending  too  much  time  planning  and  gathering 
information  and  is  unable  to  make  quick  decisions  without 
the  need  to  gather  more  information,  or  if  they  tend  to  make 
quick  decisions  without  enough  information 

THEN 

Use  CT  Skill  8:  Decide  When  to 
Seek  Information  Based  on  its 

Value  and  Cost 

Table  3.  Eight  Critical  Thinking  Skills  Listed  in  Ascending  Order  of  Difficulty 


Critical  Thinking  Skill 

Estimated  Difficulty 

Level 

CT  Skill  2:  Recognize  Gist  in  Material 

Low 

CT  Skill  4:  Generalize  from  Specific  Instances  to  Broader  Classes 

Low 

CT  Skill  1 :  Frame  the  Message 

Low  to  Medium 

CT  Skill  3:  Development  Explanation  that  Ties  Information  Together 

Medium 

CT  Skill  8:  Decide  When  to  Seek  Information  Based  on  its  Value  and  Cost 

Medium 

CT  Skill  7:  Examine  other  People’s  Perspectives 

High 

CT  Skill  5:  Use  Mental  Imagery 

High 

CT  Skill  6:  Challenge  One’s  Bias 

High 
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DEVELOPMENT  PROCESS 


Figure  3  depicts  the  five-step  process  that  was  used  to  develop  and  validate  the  training 
materials  for  (CT)  .  We  began  by  creating  an  outline  of  the  functional  specifications  for  each 
skill.  These  outlines  were  then  delineated  further,  amplified  with  examples,  and  presented  to  a 
panel  of  subject  matter  experts  (SMEs)  as  part  of  a  comprehensive  content  review.  The  SMEs 
were  instructional  developers  who  had  extensive  experience  in  developing  training  and 
educational  systems  for  the  Army.  The  feedback  loop  in  the  figure  represents  the  modifications 
to  proposed  content  that  occurred  as  a  result  of  this  review.  The  modified  content  was  then 
rendered  in  storyboard  form,  which  included  instructional  scripts,  animation,  graphics,  and 
video.  The  storyboards  were  then  given  to  the  software  developer  who  coded  each  skill  as  a 
series  of  integrated  html  pages  that  were  linked  to  a  server-resident  database.  The  pages  were 
subsequently  hosted  on  a  web  site,  and  representative  portions  of  the  training  were  given  to 
Army  personnel  during  a  series  of  four  usability  studies.  The  feedback  loop  to  Step  3  indicates 
that  comments  from  the  usability  studies  were  instrumental  in  modifying  both  the  storyboards 
(conceptual  flow)  and  the  actual  training  materials  themselves.  Below,  we  describe  each  step  in 
more  detail. 


Step  5  Usability 
Studies 


Figure  3.  Overview  of  the  five-step  development  process. 


Functional  Specifications 

We  began  (CT)2  development  by  creating  a  detailed  outline  of  the  functional  specifications 
for  each  skill.  Figure  4  provides  the  major  organizational  elements  of  those  outlines,  and  some 
example  content,  for  one  of  the  skills,  Recognize  Gist  in  Material  (Skill  2).  Appendix  B  presents 
the  completed  functional  specification  outlines  for  all  eight  skills.  For  each  skill,  a  template  was 
used  by  a  project  training-analyst  to  begin  the  process  of  creating  the  functional  specifications. 
The  organizational  elements  of  the  template  included  the  definition  of  the  skill,  a  description  of 
the  cognitive  processes  underlying  the  skill,  battle  command  tasks  that  require  that  skill,  types  of 
errors  associated  with  performing  the  skill  in  the  field,  the  tenninal  and  enabling  learning 
objectives  for  the  training  of  the  skill,  the  other  (CT)2  skills  most  closely  associated  with  the  skill 
(i.e.,  the  skill  boundaries),  the  flow  of  training  events  that  should  be  used,  and  the  types  of 
training  exercises  that  should  be  created  to  achieve  the  aforementioned  learning  objectives. 
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Skill  Definition 

This  critical  thinking  skill  is  the  ability  to  sort  through  the  details  in  a  message  (written,  graphical,  visual,  auditory, 
and/or  tabular)  and  extract  the  gist  therein. 

Description  of  Skill 

This  skill  involves  the  ability  to  sort  through  the  details  of  a  message  and  distinguish  essential  from  non-essential 
information.  It  involves  the  ability  to  generate  a  clear,  concise  statement  that  summarizes  the  gist  or  main  point  of  a 
complex  message. 

Recognizing  and  extracting  the  gist  of  a  message  is  a  useful  skill  because:  .  .  . 

Furthermore,  the  process  of  extracting  the  gist  is  a  useful  skill  because:  .  .  . 

Subcomponents  of  the  skill  are:  1)  considering  context;  2)  weighing  info,  3)  combining  &  rephrasing  info,  4) 
generating  gist  statement,  5)  reviewing  gist  statement 

Battle  Command  Task  Associated  with  Skill 

The  target  battle  command  task  for  which  this  skill  is  required  is  restating  mission  objectives  provided  by  upper 
echelons  in  order  to  write  one’s  own  mission  statements.  However,  this  skill  also  facilitates:  Better  memory  for  and 
focus  on  relevant  tasks.  Better  communication,  plus  others. 

Types  of  Errors  that  commonly  occur  when  executing  this  skill:  1)  Failure  to  distinguish  relevant  from  irrelevant 
information,  2)  Failure  to  eliminate  less  important  details  from  central  meaning  of  message,  etc. 

Learning  Objectives  for  this  Skill 

This  training  module  will  focus  on  a  set  of  learning  objectives  that  constitute  aspects  or  components  of  this  CT  skill. 
These  are:  1)  Recognizing  good  and  poor  quality  gist  statements,  2)  Identifying  errors  in  extracting  the  gist,  3) 
Understanding  the  importance  of  context,  etc. 

Skill  Boundaries 

The  most  closely  related  skill  is  CTS  1,  Framing  the  Message.  CTS  2  also  contains  elements  relating  to  CTS  6, 
Challenge  One ’s  Bias. 

Flow  of  Training 

1 .  The  training  will  begin  with  an  interactive  experience  that  shows  the  student  how  extracting  the  gist  of  a 
message  is  in  essential  to  communication  and  decision  making. 

2.  The  second  section  will  review  the  qualities  of  a  good  gist  statement,  and  common  errors  people  make  in 
extracting  the  gist 

3.  The  third  section  will  review  a  5-step  technique  to  systematically  and  accurately  extract  the  gist  from  complex 
messages.  This  section  will  walk  the  user  through  a  series  of  brief  exercises  designed  to  focus  attention  on  the 
meaning  and  purpose  of  each  step. 

4.  Then,  there  will  be  a  series  of  exercises  that  provide  an  opportunity  for  the  student  to  apply  the  technique  they 
just  learned  to  a  variety  of  military-themed  messages. 

5.  Exercises  on  applying  the  technique  to  non-text  messages,  such  as  graphics  and  tables. 

6.  Summary  of  what  students  are  expected  to  get  out  of  the  module,  and  where  it  might  be  applicable. 

7.  The  module  will  end  with  a  Post  test. 

Training  Exercises 

The  training  will  include  the  following  exercises,  broadly  described. 

1.  Beginning  scenario 

2.  Training  Set  1:  Recognizing  a  good  gist  statement 

a.  given  a  non-military  example,  classify  several  potential  “gist  statements”  by  the  type  of  error 

b.  given  a  military  message  example,  classify  several  potential  “gist  statements”  by  the  type  of  error  t 

3.  Training  Set  2:  Applying  a  5-step  technique  for  extracting  the  gist  from  complex  messages 

a.  given  a  non-military  example,  walk  through  each  of  the  five  steps  of  the  12-8-4  Technique  (read 
message,  reduce  message,  revise  key  words,  rewrite  message,  review) 

b.  Apply  the  technique  to  a  commander’s  intent  statement,  military  briefing,  and  enemy  message 

4.  Training  Set  3:  Applying  the  12-8-4  Technique  to  non-text  messages  (graphic  and  table) 

Figure  4.  Partial  example  of  a  functional  specification  for  CT  Skill  2. 
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Because  the  skills  were  developed  in  a  staggered  fashion,  the  functional  specifications  for 
CT  Skills  1-4  were  used  as  concrete  templates  for  the  later  ones.  In  addition,  the  training  analyst 
responsible  for  a  given  skill  consulted  with  the  project  director  and  other  analysts  on  a  frequent 
basis  during  creation  of  the  specifications.  These  consultations  provided  skill  development  with  a 
fresh  perspective,  and  expedited  the  creation  of  technical  content  within  a  standardized,  yet 
creative,  format.  Moreover,  as  the  project  unfolded,  the  functional  specifications  were  reviewed 
and  refined  to  ensure  that  there  was  no  overlap  in  content  coverage,  or  any  incompatibility  in 
training  flow  or  exercises. 


Content  Review 

To  ensure  that  the  training  materials  were  technically  consistent  with  Army  doctrine  and 
tactics,  and  the  strategy  compatible  with  the  Army’s  long-term  objectives,  a  panel  of  SMEs  was 
formed  to  serve  as  a  formal  review  board.  This  user  panel  consisted  of  four  retired  Army  officers 
presently  serving  as  educational  specialists  for  a  government-contracting  firm  under  contract  to 
the  Army  at  Fort  Leavenworth  to  develop  training  curricula  for  the  Command  and  General  Staff 
College  (CGSC).  All  panel  members  were  well-versed  in  instructional  design,  had  extensive 
experience  in  developing  training  curriculum  at  CGSC,  and  were  knowledgeable  of  the  goals  of 
critical  thinking  training  and  its  potential  value  for  the  Anny. 

Although  the  overall  review  was  very  positive,  the  panel  also  offered  a  number  of  useful 
suggestions  for  modifying  our  materials  and  approach.  Because  the  training  modules  adhered  to 
a  common  fonnat,  theme,  and  approach,  the  panel’s  comments  on  Skills  3-4  are  also  applicable 
to  the  other  six  skills.  Some  of  their  suggestions  for  improving  the  training  are  given  below. 

•  Consider  using  the  training  modules  in  the  context  of  classroom 
assignments  in  addition  to  using  a  distance  learning  approach. 

•  Use  current  examples  and  vignettes  to  the  extent  possible.  Although 
historical  examples  are  effective  for  making  certain  points,  the  bulk  of  the 
material  should  be  based  on  current  doctrine  and  tactics. 

•  Make  the  language  simple  and  direct,  using  graphics  and  pictorial 
representations  whenever  possible. 

•  Target  the  training  for  ROTC  students  or  junior  officers  so  that  Anny 
personnel  of  all  levels  of  experience  can  benefit  from  critical  thinking 
training. 

•  Deliver  the  training  in  the  context  of  simulated  exercises  whenever 
possible. 

•  Incorporate  expert  opinion  from  the  areas  of  learning  and  human 
information  processing  into  the  review  in  addition  to  the  Army  perspective. 

Storyboarding 

Using  the  functional  specifications  as  the  basis,  an  extended  narrative  description  of  the 
instructional  content  was  created.  This  was  done  for  each  skill.  These  descriptions  were  then 
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rendered  in  a  multi-media  storyboard.  The  storyboard,  which  contained  true  multi-media 
capabilities  (video,  sound,  animation),  was  then  rendered  within  a  web-based  environment  using 
Flash,  dynamic  HTML,  and  JavaScript.  The  steps  involved  in  developing  the  system  software  are 
described  later. 

From  the  beginning,  (CTfi  was  envisioned  as  individual,  knowledge-focused  on-line 
instruction.  The  training  analysts  expanded  each  section  of  a  given  skill’s  functional 
specifications  so  that  key  concepts  were  illustrated,  examples  sketched  out,  and  the  rudiments  of 
training  exercises  were  constructed.  To  support  this  expansion  for  each  module,  Word  files  were 
created  that  contained  narrative  descriptions  and  explanations,  supporting  graphics,  hyperlinks  to 
roll-over  (supplemental)  concepts,  and  practical  examples.  With  the  modular  Word  files  as  the 
basis,  a  multi-media  storyboard  was  then  created  for  each  skill.  This  storyboard  was  intended  to 
provide  a  very  concrete,  explicit  framework  that  would  then  be  used  by  the  software  developer  to 
create  the  web-based  content  that  would  serve  as  the  delivered  training  content. 

The  storyboard  slides  did  a  number  of  things  to  increase  student  interest  and  make  the 
training  content  appropriate  for  an  Army  audience.  First,  the  extensive  text  descriptions  were 
replaced  with  audio  voice-over  that  drastically  decreased  the  amount  of  reading  required  of  the 
user.  Second,  the  training  flow  was  revised  so  that  the  detailed  academic  points  were  simplified 
and  users  were  given  the  barebones  theoretical  information  necessary  to  appreciate  the 
importance  and  significance  of  CT  concepts.  Third,  extensive  use  was  made  of  Army  graphics 
and  military  examples  in  order  to  “draw  the  user  in”  to  the  importance  of  critical  thinking  for 
performing  critical  battle  command  tasks.  Graphics  included  maps,  tactical  sketches,  satellite 
images,  photos,  and  other  pictorial  information.  In  creating  the  narrative  voice-over,  we  were 
mindful  of  the  beneficial  effects  on  learning  that  comes  from  a  judicious  integration  of  audio 
narration  with  graphic  illustration  (Mayer,  2001).  When  completed,  the  PowerPoint  files  were 
quite  large  (>10MB),  owing  to  the  liberal  use  of  graphics,  audio,  and  video.  The  files  were  then 
sent  to  an  ftp  site,  which  the  software  developer  then  used  as  the  basis  for  his  Flash/JavaScript/ 
dynamic  HTML  programming. 

In  creating  the  storyboards,  significant  effort  went  to  transfonning  the  theoretical  principles 
of  critical  thinking  into  practical  skills.  Content  was  rendered  in  a  PowerPoint  form  that 
exploited  the  learning  benefits  of  audio  narration,  drag-and-drop  interactivity,  visual  effects 
(transitions,  fade-ins,  pop-ups),  and  high-resolution  graphics.  Practice  was  then  embedded  in  the 
training  in  a  scenario-based,  interactive  multimedia  interface.  This  interactive  environment 
compels  the  user  to  make  practical  judgments  in  a  credible  environment.  After  the  user  sees  the 
sensible  use  of  the  Skill,  the  web-based  training  presents  enough  of  the  theoretical  underpinnings 
to  cement  the  principles  behind  the  learning. 

Software  Development 

Due  to  the  ease  with  which  relatively  complex  user  interactions  can  be  developed  using  the 
browser-based  technologies  of  HTML,  CSS  (cascading  style  sheets),  and  JavaScript 
manipulation  of  the  DOM  (document  object  model),  these  three  technologies  have  collectively 
come  to  be  known  as  dynamic  HTML  (DHTML),  and  constitute  the  software  for  the  training. 
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In  addition  to  DHTML,  a  variety  of  other  software  tools  were  used  to  craft  the  (CT)  training 
environment.  These  included:  (1)  Macromedia  Flash™  for  creating  animations  and 
synchronizing  audio;  (2)  Adobe  Photoshop™  for  manipulating  and  compressing  graphic  files; 

(3)  Macromedia  Dreamweaver™  for  creating  HTML  source  code;  (4)  IBServer  (a  Sourceforge 
project  distributed  under  the  GPL  license)  as  a  Windows™-based  testing;  and  (5)  Sapien’s 
PrimalScript™  integrated  development  environment  for  Windows™  scripting  languages. 

From  a  user  interface  standpoint,  a  host  of  functional  capabilities  and  visual  formats  were 
created  to  support  the  functional  specifications  required  for  the  eight  skills.  In  some  cases,  the 
capabilities  were  developed  using  the  services  offered  by  DHTML  and  supported  by  the  Internet 
Explorer  browser.  Other  times,  new  tools  and  display  formats  were  designed.  To  illustrate  the 
strategy  used  for  user  interface  design,  Figure  5  depicts  a  display  screen  from  one  of  the  practical 
exercises  in  Skill  1,  Frame  the  Message.  The  “look  and  feel”  of  this  screen,  and  the  associated 
functionality,  is  fairly  typical  of  the  instructional  content  and  exercises  that  populate  our  training 
system. 


Practice  Exercise  #1 

You  are  the  executive  officer  of  a  military  intelligence  battalion.  It  is  Friday  morning  and  you  are  arriving  at  your 
office.  Last  night  you  received  a  phone  call  from  your  HHQ  company  commander.  She  told  you  that  one  of  her 
soldiers  -  Terry  Jensen  -  was  involved  in  a  domestic  abuse  incident  in  military  quarters.  A  sworn  statement  from  the 
soldier  in  question,  gathered  by  the  military  police  who  responded  to  the  incident  is  on  your  desk  when  you  arrive: 

Click  on  WORKSPACE  to  begin  working.  |  WORKSPACE 

In  the  table  below,  analyze  the  message  by  classifying  each  of  its  components  into  the  categories  of  who,  what, 
why,  when,  where  and  how.  If  you  drag  your  mouse  over  the  message,  you'll  see  that  some  of  its  components 
become  highlighted.  Drag  each  component  to  the  appropriate  place  in  the  table  below.  Each  row  of  the  table  shoulc 
represent  a  single  idea,  usually  a  sentence,  but  not  always. 

If  you  want  to  move  a  component  once  you  have  dragged  it  to  a  slot,  highlight  it  then  click  the  "DELETE  WORD" 
button.  When  you  are  done  with  one  relationship  section,  click  the  "PUT  IN  RELATIONSHIP  TABLE"  button  to  receive 
feedback  and  move  on  to  the  next  relationship. 


Our  company  was  deployed  until  today  to  support  a  Division  Exercise.  We  weren’t  supposed  to  return  until  next 
week.  I  came  home  last  night  and  discovered  my  spouse  with  someone  else  in  my  bedroom.  They  were  both  half 
dressed.  I  was  angry  enough  that  I  don’t  remember  everything.  But  I  do  remember  screaming  and  throwing 
whatever  I  could  find  in  their  general  direction.  I  can  remember  the  stranger  running  past  me  to  leave  and  I 
can  remember  that  one  of  the  objects  I  grabbed  and  threw  hit  Pat,  my  spouse,  on  the  face.  When  I  saw  what  I 
had  done,  I  stopped  throwing  stuff.  However,  I  was  still  very,  very  angry.  I  left  the  house  and  went  to  a 
bar.  I  was  having  a  beer  when  the  HPs  picked  me  up. 


Relationship  1  of  14 


Who 

What 

Why 

When 

Where 

How 

|  DELETE  WORD  |  [  PUTIN  RELATIONSHIP  TABLE  | 


Figure  5.  Example  of  the  user  interface  of  (CT)2. 

As  can  be  seen,  the  text  in  the  upper  part  of  the  screen  presents  the  student  with  a  “scenario” 
that  is  intended  to  structure  his/her  critical  thinking  in  the  exercise.  The  scenario  usually 
indicates  the  role  the  student  is  to  play  (Executive  Officer  in  a  military  intelligence  battalion), 
along  with  contextual  and  background  information  that  sets  the  scene  (receiving  a  phone  call 
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from  the  Commanding  Officer  (CO)  about  a  domestic  abuse  incident).  Below  that  is  presented 
some  instructions  for  interacting  with  the  instructional  material  that  is  about  to  follow. 

A  message  then  appears  that  the  student  must  classify  using  various  drag-and-drop  tools. 
Mousing  over  the  text  will  select  it,  where  drag-and-drop  functionality  is  invoked  for  moving 
words  and  phrases  into  a  relationship  table.  Buttons  at  the  bottom  of  the  screen  let  the  student 
operate  on  the  infonnation  at  a  more  macro  level.  The  interplay  between  message  analysis, 
classification,  and  selection/deletion  lies  at  the  heart  of  our  practical,  hands-on  approach  to 
critical  thinking.  While  other  exercises  have  other  tools  (e.g.,  selecting  symbols  on  a  map, 
moving  cells  in  a  table),  the  same  basic  philosophy  applies.  As  shown  by  the  usability  studies 
described  in  the  next  section,  this  graphics-oriented,  interactive-focused  approach  to  training  was 
well  received  by  learners. 
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FORMATIVE  EVALUATIONS:  USABILITY  INVESTIGATIONS 


Due  to  scheduling  constraints,  it  was  not  possible  to  subject  all  eight  CT  skills  to  a  compre¬ 
hensive  usability  review.  Consequently,  fonnative  evaluations  were  conducted  on  a  sample  of 
training  components.  Nevertheless,  we  believe  this  was  a  legitimate  strategy  since  the  pretest, 
training  module,  and  posttest  share  many  features  across  skills,  including  user  interface,  fonnat, 
framework,  logic,  and  functionality.  Accordingly,  user  reaction  to,  and  comments  on,  any  subset 
of  pretest/training  modules/posttest  will  apply  to  the  entire  set.  The  results  of  the  usability  tests 
on  these  samples  were  then  used  to  inform  the  other  tests  and  modules. 

During  April  -  September  2005,  four  usability  tests  were  conducted.  The  sites  were  Fort 
Riley,  KS;  Fort  Hood,  TX;  Fort  Lewis,  WA;  and  the  63rd  Regional  Readiness  Command  (RRC) 
Los  Alamitos,  CA.  Table  4  provides  an  overview  of  the  four  usability  tests.  As  can  be  seen,  the 
usability  tests  covered  the  pretests  for  Skills  1-4  as  well  as  the  training  modules  associated  with 
Skills  2  and  4.  A  total  of  19  Army  personnel  served  as  subjects,  ranging  in  rank  from  lieutenant 
to  major.  In  addition,  one  participant  was  an  ROTC  cadet. 


Table  4.  Descriptive  Overview  of  the  Four  Usability  Tests 


Location 

Date 

Participants 

Materials  Assessed 

Fort  Hood 

26-29  April  05 

6/  2  majors  (Artillery),  2 
lieutenants  (Finance),  2 
lieutenants  (Medical  coips) 

Pretest,  Skill  3  ( Develop 
Explanation );  Pretest,  Skill  4 
( Generalize  from  Specific)', 

Training  Module  Skill  4 

Fort  Riley 

6-10  June  05 

5/  1  majors  (Signal  Coips),  1 
captains  (Chemical  Corps),  3 
lieutenants  (Ordnance,  Signal, 
Chemical  Corps) 

Pretest,  Skill  1  ( Frame  Message)', 
Pretest,  Skill  2  ( Recognize  Gist) 

Fort  Lewis 

26-28  July  05 

5/  2  captains  (Armor,  Artillery),  2 
lieutenants  (Signal  Coips,  Arty), 

1  cadet 

Training  Module  Skill  4 
(< Generalize  from  Specific) 

Los  Alamitos 
Army  Reserve 
Base 

10-11  Sept  05 

3/  2  lieutenants  (Medical),  1 
captain  (Engineer) 

Training  Module  Skill  2 
{Recognize  Gist) 

A  detailed  description  of  each  usability  evaluation  is  available  in  the  technical  memoranda 
published  on  each  investigation  (Mautone,  2005;  Holder  &  Fischer,  2005;  Holder,  Campsey,  & 
Fischer,  2005a;  Holder,  Campsey,  &  Fischer,  2005b).  Though  different  training  materials  were 
reviewed  in  the  four  studies,  a  common  method  and  procedure  was  used,  as  described  below. 

General  Method 

All  personnel  participated  as  part  of  a  research  support  request  scheduled  either  during  a 
reserve  training  weekend  or  as  part  of  the  Army’s  Umbrella  Week  program.  All  were  told  their 
participation  was  voluntary.  For  each  investigation,  the  training  module  and/or  pretest/posttest 


21 


was  programmed  in  an  HTML  presentational  format  and  was  run  via  an  Internet  Explorer  web- 
browser  from  a  laptop  computer.  The  evaluation  focused  on  three  main  areas: 


(1)  Overall  usability:  Were  users  able  to  easily  navigate  the  training  program?  Were  the 
mechanics  of  the  exercises  (e.g.,  the  drag-and-drop  procedure)  clear  and  easy  to  use?  Did 
they  have  any  comments  or  suggestions  regarding  the  format  and  layout  of  the  module? 

2)  Instructional  methods:  Was  the  wording  easy  to  follow?  Were  the  lessons,  examples, 
and  exercises  engaging?  Did  they  feel  like  they  learned  something?  Was  the  module 
worthwhile?  Could  they  see  others,  perhaps  those  less  experienced,  gaining  useful 
knowledge  and  skills  from  the  training  module? 

3)  Content:  Did  the  military  examples  and  exercises  seem  realistic  and  engaging?  Did 
users  understand  and  agree  with  the  feedback?  Did  the  scenarios  require  too  much 
specialized  knowledge?  Was  the  terminology  unclear  -  either  for  the  users  or  perhaps  for 
others  who  have  less  experience? 

Each  investigation  utilized  an  assortment  of  assessment  techniques,  including 
questionnaires,  observation,  a  think-aloud  protocol,  and  interviews.  The  variety  of  data  collection 
methods  afforded  gathering  subjective  opinion  of  the  materials  as  well  as  objective  observations 
of  the  training  materials’  usability. 

Two  questionnaires  were  administered  using  a  laptop  computer.  Examples  of  each  are 
shown  in  Appendix  C.  The  first  was  a  demographic  instrument  that  asked  about  participants’ 
military  experience,  educational  background,  Internet  usage,  and  experience  with  and  attitudes 
toward  web-based  training.  The  second  questionnaire  asked  participants  to  rate  and  comment  on 
general  components  of  the  training  module  -  such  as  the  usability,  clarity,  logic,  usefulness,  and 
length  of  the  training  program  as  a  whole.  Participants  were  also  asked  to  answer  questions  about 
specific  components,  such  as  the  usefulness  and  clarity  of  the  training  exercises,  feedback, 
lessons,  and  introductory  sections.  Most  of  the  questions  used  five-point  Likert  scale  ratings 
(l=strongly  disagree,  5=strongly  agree)  were  used. 

The  experimenter  directly  observed  participant  interaction  with  the  materials  and  took  notes 
on  their  behaviors,  comments,  and  questions.  Participants’  on-screen  activities  were  recorded  at 
6-second  intervals  using  Win-Spy  Software  8.3  Pro  screen-capturing  software.  User  comments 
were  also  tape-recorded. 

Participants  were  instructed  to  verbalize  their  thoughts,  or  “think-aloud,”  as  they  completed 
each  training  item.  This  technique  has  been  shown  to  be  highly  effective  for  uncovering  the 
purpose  behind  participants’  actions,  as  well  as  identifying  confusing  language  and  issues  with 
navigation  (Cordingley,  1989;  Kirwan  &  Ainsworth,  1992).  The  “think-aloud”  data  was  recorded 
on  audiotape  and  was  supplemented  with  the  experimenter’s  notes. 

The  experimenter  also  used  free-form  interviews  to  obtain  further  data  and  to  clarify  issues 
observed  during  task  performance.  Interview  procedures  depended  on  the  specific  issue  in 
question.  Questions  that  did  not  interfere  with  participants’  natural  performance  were  asked  as 
they  worked  through  the  material.  Other  interview  questions  were  noted  and  asked  after 
participants  had  completed  their  review  of  the  material  being  evaluated.  As  needed  for 


22 


clarification,  the  experimenter  redisplayed  particular  content  to  participants  or  provided  paper 
copies  of  the  material.  Issues  discovered  with  the  first  few  participants  of  each  investigation  were 
broached  with  subsequent  participants  to  solicit  a  variety  of  opinions. 

General  Results 

Specific  results  for  each  training  component  are  reported  in  the  technical  memoranda  cited 
previously.  Below,  we  present  examples  of  the  general  findings  that  characterized  most  or  all  of 
the  formative  evaluation  studies.  To  facilitate  presentation,  the  general  findings  are  organized 
according  to  general  usability,  instructional  methods,  appropriate  user  population,  and  technical 
content. 

Overall  Usability 

Navigation  and  Usability.  While  all  participants  were  generally  able  to  navigate  through 
the  program  with  ease,  several  expressed  a  desire  for  a  page  counter  or  progress  bar  to  indicate 
the  amount  of  material  remaining.  One  participant  suggested  building  a  navigation  window  that 
provides  an  outline  of  the  module,  allowing  users  to  go  back  and  forth  between  pages  previously 
visited.  To  address  this  issue,  a  “past  pages”  link  was  incorporated  into  each  training  module, 
which  allows  users  to  revisit  previously  seen  pages  as  well  see  a  listing  of  all  pages  in  the 
module. 

Various  minor  navigational  issues  were  identified  that  were  specific  to  each  component 
tested,  e.g.,  confusion  about  which  button  to  press,  inability  to  find  a  link,  etc.  Hence,  these 
navigational  issues  were  corrected  in  the  final  version  of  the  program,  and  those  corrections  were 
transferred  to  all  standard  training  modules.  For  example,  the  CT  logo  designed  for  the  training 
was  removed  from  every  training  module  page  except  the  first  page  of  each  because  some 
reviewers  thought  it  was  a  link.  Based  on  participants’  comments,  additional  changes  were  made 
to  the  formatting  (e.g.,  bolding,  color,  italics)  of  the  text  in  the  training  components  evaluated  to 
direct  users’  attention  to  appropriate  navigational  cues,  and  these  changes  were  made  throughout 
all  training  components.  Participants  also  requested  the  ability  to  cut  and  paste  their  own  typed- 
in  text  to  reduce  typing  effort.  This  capability  was  provided  in  all  modules  in  their  final  versions. 

Understandability.  The  reading  level  required  for  each  training  component  was  evaluated 
with  standard  word  processing  tools.  Changes  were  made  to  any  text  that  exceeded  a  ninth  grade 
reading  level,  and  such  modifications  were  made  prior  to  usability  testing.  Thus,  materials 
presented  to  participants  in  the  usability  evaluation  were  at  a  ninth  grade  reading  level  and  users 
generally  thought  that  they  were  understandable.  However,  there  were  elements  of  each  training 
component  that  was  evaluated  that  confused  particular  participants;  modifications  were  made  to 
the  final  versions  to  address  those  confusing  elements. 

Length.  All  participants  thought  the  pretests  and  posttests  lengths  were  “just  about  right.” 
Evaluators  of  the  Skill  2  training  module  thought  that  this  module  was  a  little  too  long  and 
repetitive.  The  length  of  this  module  was  reduced  in  the  final  version,  although  it  remains  the 
longest  module  incorporated  in  (CT)2. 
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Graphics.  Skill  4  ( Generalize  from  specific  instances )  tests  and  training  made  extensive  use 
of  photographic  stimuli  for  presenting  examples  and  illustrating  principles.  Several  participants 
expressed  a  desire  to  zoom  in  and  out  of  the  pictures  to  pennit  detailed  aspects  of  the  picture  to 
be  more  closely  scrutinized.  For  some  graphics,  then,  a  zoom  function  was  added.  However, 
rather  than  providing  a  zoom  function  for  all  graphics,  some  graphics  were  enlarged  to  address 
this  issue.  All  participants  responded  that  the  viewing  time  for  graphics  was  just  about  right. 

In  addition,  participants  made  a  variety  of  suggestions  to  enhance  usability.  The  most 
frequently  occurring  suggestion  was  to  shorten  the  lengths  of  some  of  the  pages.  Thus,  page 
length  and  amount  or  required  reading  was  kept  to  a  minimum  in  each  training  module.  Also, 
participants  found  a  number  of  typos,  missing  words,  misspellings,  overly  long  sentences  and 
paragraphs,  awkward  phrasing,  and  ambiguous  words.  These  deficiencies  were  corrected  as 
discovered,  and  the  modifications  were  incorporated  into  the  next  formative  evaluation 
investigation. 

Instructional  Methods 

As  noted  previously,  participants  favored  web  pages  in  training  modules  that  minimized  the 
amount  of  required  reading,  thus,  an  instructional  method  was  adopted  that  utilized  the 
presentation  of  informational  material  in  audio  fonn  in  lieu  of  text  that  had  to  be  read.  Each  of 
the  final  versions  of  the  standard  training  modules  presents  audio  files  that  are  linked  to  relevant 
graphics  presented  on  the  screen.  In  some  cases,  the  student  is  presented  with  both  textual 
material  and  audio  files  if  the  infonnation  requires  referral  or  is  complex. 

Participants  also  had  difficulty  following  training  exercises  that  included  multiple  steps,  i.e., 
were  complex.  Thus,  many  of  the  exercises  that  can  be  found  in  the  final  versions  of  the  standard 
training  modules  are  simpler  than  they  were  originally  designed.  To  simplify  the  exercises,  the 
number  of  steps  required  for  their  completion  was  reduced. 

All  exercises  incorporated  feedback  after  each  user  response.  However,  participants  had 
varying  opinions  about  the  amount  and  kind  of  feedback  they  favored.  For  example,  in  Skill  2 
(. Recognize  Gist),  some  participants  liked  having  multiple  perspectives  pointed  out  in  the 
feedback  section.  On  the  other  hand,  other  participants  found  this  kind  of  feedback  problematic 
because  there  was  no  “right  answer”  and  differing  opinions  are  vague  and  difficult  to 
comprehend.  While  the  majority  of  participants  indicated  that  the  feedback  was  infonnative  and 
clear,  an  effort  was  made  to  limit  the  use  of  multiple  perspectives  (multiple  right  answers)  in 
feedback.  In  some  standard  training  modules  this  kind  of  feedback  is  still  used  because  the  nature 
of  the  exercise  itself  does  not  lend  itself  to  a  single  clear  answer.  In  these  cases,  the  exercise 
experience  was  given  a  higher  priority  than  the  feedback. 

Although  (CT)  was  designed  to  be  individual  training,  the  investigations  were  conducted 
with  groups.  Several  participants  noted  the  training  value  of  reviewing  the  material  as  part  of 
group,  where  student  discussion  and  instructor  feedback  clearly  facilitated  the  students’  learning 
of  key  concepts.  Consequently,  group  learning  should  be  considered  as  a  potentially  valuable 
educational  tool  for  future  implementations  of  (CT)2. 
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Appropriate  Student  Population 

Most  users  thought  that  the  training  could  be  effectively  provided  to  students  with  less 
experience  than  captains  and  majors.  It  was  believed  possible  and  desirable  to  deliver  the 
training  to  third  and  fourth  year  West  Point  cadets  and  ROTC  students.  Most  participants 
understood  the  concepts,  principles,  and  illustrations  of  the  CT  skills  that  were  presented.  They 
took  this  to  mean  that  the  material  could  also  be  presented  to  student  populations  that  had  less 
experience  than  themselves. 

Technical  Army  Content 

One  of  the  most  highly  rated  aspects  of  the  system  concerned  the  Army  contexts  and 
scenarios.  Participants  thought  that  setting  the  exercises  in  realistic  Army  scenarios  gave 
credibility  to  the  training.  Indeed,  participants  uniformly  endorsed  the  concrete  military 
examples.  However,  several  users  indicated  that  they  would  prefer  to  see  such  examples  right  in 
the  beginning  of  the  training,  to  help  users  see  the  relevance  of  the  topic.  Hence,  the  final 
versions  of  the  standard  training  modules  utilize  Army  contexts  and  scenarios  early  in  the 
training.  In  short,  the  Anny  contexts  and  scenarios  were  seen  as  an  asset,  and  participants  found 
the  technical  language  easy  to  understand.  All  modules,  pretests,  and  posttests  incorporate 
realistic  Anny  scenarios. 

While  there  were  some  comments  critical  of  the  particular  wording  of  terms,  there  was  little 
disagreement  regarding  the  technical  veracity  of  the  information  being  presented.  However,  a 
few  participants  made  comments  about  certain  phrases  and/or  individual  pictures.  The  technical 
memos  (Mautone,  2005;  Holder  &  Fischer,  2005;  Holder,  et  ah,  2005a;  Holder,  et  ah,  2005b)for 
each  usability  evaluation  give  detailed  descriptions  of  feedback  provided  by  users.  For  each 
investigation,  user  comments  were  distilled  into  a  three-column  table  corresponding  to  item, 
issue,  and  suggested  improvement.  An  example  of  such  a  table,  taken  from  the  evaluation  of 
Skill  4  at  Fort  Lewis  (Holder  et  ah,  2005b)  is  shown  in  Table  5.  Due  to  space  constraints,  this 
table  is  only  a  partial  representation  of  the  entire  data  set  collected  at  this  site. 

Conclusions 

Taken  together,  the  results  of  the  four  usability  evaluations  found  that  (CT)"  was  generally 
well  designed,  although  many  specific  improvements  were  needed.  Using  the  data  illustrated  in 
Table  5,  the  desired  changes  were  presented  to  the  system  programmer  immediately  following 
each  usability  evaluation.  The  suggested  improvement  column  (column  3  in  Table  5)  was  the 
means  by  which  the  changes  required  were  communicated.  The  formative  evaluations  proved  to 
be  a  very  effective  and  efficient  means  for  improving  the  overall  quality  and  relevance  of  the 
computer-based  training  system. 
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Table  5.  Example  of  the  Data  Collected  from  the  Usability  Evaluations 


Skill  4  Generalize  from  Specific  Instances  | 

Item 

Issue 

Suggested  Improvement 

Overall 

Pictures  have  a  menu  bar  when  your  mouse 
is  over  it  that  covers  up  important  features 

Disable  the  menu  bar 

Gut  Check  ex.  1 

Participants  did  not  realize  they  were  under 
time  pressure  and  lost  valuable  time 

Put  the  timing  statement  in  bold  and  then  flash  an 
in-between  screen  that  tells  them  they  only  have 

30  seconds 

Gut  Check  Ex  1 

Timer  counts  up 

Make  timer  count  down 

Gut  Check  Ex.  1 :  2nd  response 
box 

Some  participants  needed  a  reminder 
definition  of  the  feature  types,  especially 
contextual  features 

Include  roll-overs  for  the  feature  column  heads 
with  a  definition  or  example 

Gut  Check  Ex.  1 :  2nd  response 
box 

Order  of  the  feature  types  is  inconsistent 
with  later  exercises 

Switch  order  to  physical,  contextual,  then 
psychological 

Gut  Check  Ex.  1 :  2nd  response 
box 

When  the  page  scrolls  it  cuts  off  the  picture 
although  it  does  not  have  to 

Fix  placement  of  automatic  scroll 

Gut  Check  Ex.  1 :  Photo 

Several  participants  focused  on  the 
prominence  of  black  clothing  possibly  as  a 
sign  of  mourning. 

Consider  further  for  inclusion  in  the  feedback 

Gut  check  Ex.  1:  Comparison 
page 

Participants  did  not  notice  the  compare 
statement 

Make  the  comparison  statement  larger  and  in 
bold  to  grab  attention 

Devil  in  the  Details  Ex.  2 

Participants  could  not  find  the  6th  kidnapper 
unless  you  assume  it  is  a  driver 

Change  feedback  to  say  at  least  5  and  likely  6 
with  the  driver 

Illustrative  scenario 

The  word  palpable  was  not  known 

Change  to  “strong” 

Defining  Features  page 

Under  the  tank  comparison  text  add  some 
pictures  of  tanks 

Add  pictures  and  words  asking  which  of  these  are 
tanks,  BMPs,  Russian  T-55,  T-80,  Panzer,  then 
Stryker,  BTR,  consider  if  they  need  to  actively 
check  yes  or  no  to  make  it  interactive.  We  can 
then  can  have  the  terrorist  paragraph  under  the 
screen  break  to  divide  the  reading  (see  above) 
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HARDWARE  AND  SOFTWARE  DESCRIPTION  AND  REQUIREMENTS 

Hardware  Requirements 

All  (CT)  software  was  developed  assuming  that  the  training  would  be  portrayed  on  mid¬ 
range  desktop  or  laptop  computers.  In  particular,  the  hardware  requirements  include  (1) 
broadband  connection  to  the  Internet  in  which  Internet  Explorer  (IE)  version  6.0  or  greater  is 
installed;  (2)  a  video  card  capable  of  16-bit  color  at  a  minimum  resolution  of  1024  X  768  pixels; 
(3)  17-inch  color  monitor  (15-inch  on  a  laptop);  (4)  hard  disk  with  20  -  30  MB  of  free  space;  and 
(5)  sound  capability. 


Software 

(CT)"  is  a  SCORM  compliant  web-based  training  package.  The  content  structure  is  primarily 
made  up  of  HTML  pages.  However,  other  media  types  used  in  the  training  include: 

•  Javascript  fdes  (.js) 

•  CSS  Style  fdes  (.css) 

•  Adobe  Flash  files  (.swf) 

•  image  files  (.jpg  and  .gif) 

•  audio  files,  (.wav  and  .mp3) 

•  video  files  (unpeg) 

•  PDF  files  (.pdf) 

All  programming,  SCORM  and  otherwise,  is  handled  on  the  client  side  via  Javascript.  No 
server  side  programming  (e.g.  ASP,  PHP,  CGI)  is  used  in  the  training  to  ensure  compatibility 
with  a  wide  variety  of  server  configurations.  All  database  capabilities  of  (CT)"  are  handled 
through  SCORM;  therefore,  no  specific  database  vendor  is  required  for  its  use. 

At  the  SCORM  level,  (CT)2  comprises  eight  skill  directories  (e.g.  Skill  1,  Skill  2),  as  well  as 
directories  containing  global  reference  files  that  are  used  by  all  Sharable  Content  Objects  (SCOs) 
during  runtime.  Each  skill  directory  is  in  turn  broken  down  into  individual  SCOs,  housed  in 
separate  folders.  All  SCORM  manifest  files  are  included  in  the  root  level  of  the  SCOs.  As  a  rule, 
initiation  of  each  SCO  is  handled  by  the  index  page  of  its  directory. 

Consistent  with  SCORM  compliancy  standards,  the  training  package  is  designed  to  run  on 
an  LMS.  (CT)2  is  currently  being  tested  on  Moodle,  which  is  an  open  source  LMS.  However,  any 
2004  SCORM  compliant  LMS  should  be  compatible  with  each  of  the  training  and  assessment 
components  of  (CT)2. 

All  administrative  features  of  the  training  can  be  accessed  through  an  LMS.  The  following 
features  can  be  tracked  through  the  LMS  database: 

•  SCO  initiation 

•  SCO  page  to  page  movement 

•  SCO  test  scores 

•  SCO  time  on  test 
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SUMMATIVE  EVALUATIONS 


2 

To  evaluate  whether  (CT)  increases  CT  skill,  two  summative  evaluations  were  conducted. 
The  first  evaluation  was  designed  to  assess  the  learning  achieved  through  one  training  module 
compared  to  a  control  group.  This  investigation  was  conducted  at  Fort  Riley,  Kansas.  The  second 
investigation  assessed  whether  the  extended  version  of  the  training  provides  additional  learning 
over  and  beyond  the  gains  in  skill  achieved  through  the  standard  training  module.  This 
investigation  was  conducted  with  volunteer  ROTC  students  from  the  University  of  California, 
Santa  Barbara. 


Investigation  1 

The  central  objective  of  this  investigation  was  to  determine  the  learning  effect  of  training 
provided  by  (CT)"  on  a  particular  CT  skill.  A  comparison  was  made  between  Soldiers  that 
received  (CT)2  training  and  a  control  group  that  did  not  receive  (CT)2  training.  The  training 
component  of  one  CT  skill,  Skill  8:  Decide  When  to  Seek  Information  Based  on  Its  Value  and 
Cost,  was  used  in  the  investigation.  The  central  hypothesis  tested  was  that  participants  who 
complete  (CT)"  training  on  one  particular  CT  skill  will  display  superior  performance  on  that  skill. 
Post  and  pre  training  assessments  were  taken  on  both  the  experimental  and  control  groups  to 
control  for  potential  variation  in  pre-training  ability. 

Method 

Participants.  Twenty-one  male  Soldiers  participated  in  the  evaluation.  Their  ranks  included 
1st  lieutenant  (n  =  5),  2nd  lieutenant  (n  =  4),  captain  (n  =  1 1),  and  major  (n  =  1).  Eleven 
participants  were  randomly  assigned  to  the  no-training  condition  and  10  were  randomly  assigned 
to  the  training  condition.  Participants  were  originally  tasked  to  attend  the  session  as  part  of  a 
scheduled  data  collection  activity  at  Fort  Riley,  Kansas.  Upon  arrival  participants  were  informed 
that  their  participation  was  voluntary.  All  participants  provided  informed  consent. 

Materials.  The  materials  used  in  this  investigation  were  a  subset  of  the  (CT)"  training 
materials.  Specifically,  the  materials  were  taken  from  the  training  module  for  Skill  8:  Decide 
When  to  Seek  Information  Based  on  Its  Value  and  Cost.  Participants  were  first  given  a  pretest  to 
assess  their  ability  on  this  skill.  This  pretest,  which  was  developed  specifically  for  this 
evaluation,  was  essentially  a  parallel  version  of  the  post-test  component  of  the  module. 
Participants  assigned  to  the  experimental  condition  then  completed  the  training  module  for  Skill 
8.  Finally,  all  participants  were  given  the  posttest  for  the  module,  assessing  the  participants’ 
ability  to  decide  when  to  seek  information  based  on  its  value  and  cost.  A  brief  demographic 
questionnaire  was  also  used.  The  training  items  were  programmed  in  an  HTML  to  allow  for  the 
intended  web-based  delivery.  The  materials  were  run  via  an  Internet  Explorer  web-browser  from 
a  laptop  computer. 

Procedure.  Upon  arrival,  participants  were  greeted,  and  were  asked  to  select  a  card  that 
randomly  assigned  them  to  the  training  or  no-training  group.  Then,  they  were  given  a  brief 
overview  of  the  project,  its  purpose,  a  description  of  the  data  collection  procedures,  and  a 
discussion  of  how  the  data  would  be  used.  Next,  the  participants  read  and  completed  the  consent 
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form  and  a  demographic  questionnaire.  After  all  questions  were  answered,  all  2 1  participants 
completed  the  Skill  8  pretest.  After  completing  the  pretest,  participants  assigned  to  the  no¬ 
training  group  took  a  15-minute  break,  and  then  completed  the  Skill  8  posttest.  After  completing 
the  Skill  8  pretest,  the  experimental  group  completed  the  Skill  training  module,  followed  by  the 
Skill  8  posttest.  They  then  completed  a  posttest  questionnaire  soliciting  their  opinions  about  the 
training.  The  experimental  group  was  allowed  to  take  breaks,  as  desired,  while  completing  the 
training. 

Results 

Participants’  posttest  scores  were  subjected  to  a  One-Way  Analysis  of  Covariance 
(ANCOVA)  that  examined  differences  in  post-test  scores  between  the  training  and  no-training 
groups,  using  participants’  pretest  scores  as  a  covariate  to  control  for  baseline  skill.  The  training 
group  performed  significantly  better  than  the  no  training  group  on  the  posttest  F(\,  18)=  10.121, 
p  <  .01  =  59.922.  The  pretest  and  posttest  means  (unadjusted  and  adjusted  for  pretest)  are  listed 
in  Table  6. 

Table  6.  Pretest  and  Posttest  Means  and  Standard  Deviations  by  Group  for  Investigation  1 


Group 

Pretest  Mean  (SD) 

Posttest  Mean  (SD) 

Posttest  Mean  (SD) 

(unadjusted) 

(adjusted) 

Training  (N=10) 

50.5  (11.2) 

64.0  (8.4) 

61.4(8.4) 

No  Training  (N=l  1) 

42.3  (13.8) 

47.7(12.3) 

50.1  (12.3) 

Discussion 

The  results  indicate  that  completing  the  Skill  8  training  module  increased  a  participant’s 
skill  at  deciding  when  to  seek  infonnation  based  on  its  value  and  cost.  These  findings  suggest 
that  (CT)2  is  effective  at  increasing  learning  on  this  particular  skill.  Because  the  training  and 
testing  components  of  each  of  the  eight  skills  follow  a  similar  pedagogy  and  presentation  format, 
one  might  expect  that  each  training  module  would  produce  a  similar  learning  effect.  However, 
further  research  would  be  necessary  to  test  that  assumption. 

Investigation  2 

The  central  objective  of  the  second  summative  evaluation  was  to  determine  if  an  extended 
training  module  provides  greater  learning  than  the  standard  training  for  a  particular  CT  skill.  The 
extended  version,  of  course,  requires  a  greater  time  commitment  than  does  the  standard  version. 
Previous  studies  had  shown  that  the  extended  versions  took  around  10  to  13  hours  to  complete; 
whereas  the  standard  training  modules  were  designed  to  take  no  more  than  2  hours.  Hence,  the 
investigation  sought  to  determine  if  the  greater  time  cost  was  worth  the  investment,  i.e.,  paid  off 
in  greater  learning.  This  investigation  was  accomplished  with  the  assistance  of  the  University  of 
Califomia-Santa  Barbara  (UCSB),  Military  Sciences  Department  ROTC  Surfrider  Battalion. 
Cadets  and  cadre  from  the  ROTC  department  volunteered  to  participate  in  the  investigation.  The 
central  hypothesis  tested  was  that  participants  who  complete  an  extended  training  version  of  a 
particular  CT  skill  will  display  superior  perfonnance  on  that  skill  compared  to  participants  who 
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complete  the  standard  training  module.  The  research  also  evaluated  participant’s  subjective 
preferences  between  the  two  versions  of  the  training. 

Method 

Participants.  Twenty-one  participants  completed  the  experiment.  Eight  completed  the 
extended  version  of  the  training  and  13  completed  the  standard  version.  Of  these  participants,  5 
were  cadre  (instructors)  and  16  were  cadets  (students).  Eighteen  males  and  3  females  comprised 
the  participant  sample.  All  participants  provided  informed  consent.  For  each  participant  who 
completed  the  standard  version  of  the  training,  a  contribution  of  $25  was  made  to  the  UCSB 
ROTC  Surfrider  Battalion’s  Morale  and  Welfare  fund.  For  each  participant  who  completed  the 
extended  version  of  the  training,  $75  was  contributed  to  the  fund. 

Facilities  and  Equipment.  The  experiment  was  conducted  at  the  UCSB’s  Military  Science 
computer  laboratory.  Six  laboratory  computers  and  3  laptops  were  used  to  display  the  training 
and  other  materials  in  the  experiment. 

Training  Materials.  Four  components  of  the  Skill  4  ( Generalize  from  Specific  Instances  to 
Broader  Classes)  training  module  were  used  as  materials  in  this  investigation.  The  pretest 
component  of  Skill  4  was  used  to  assess  the  participants’  pre-training  ability  regarding  this  CT 
skill.  The  extended  training  version  and  the  standard  training  version  of  the  Skill  4  module  were 
also  used.  Following  completion  of  the  either  the  extended  or  standard  training  version, 
participants  were  also  asked  to  complete  the  posttest  component  of  Skill  4  as  a  post  training 
assessment  of  their  skill  level.  The  training  items  were  programmed  in  HTML  to  allow  for  the 
intended  web-based  delivery.  The  materials  were  run  via  an  Internet  Explorer  web-browser  from 
a  laptop  computer. 

Procedure.  Prior  to  the  investigation,  visits  were  made  to  the  UCSB  Military  Science 
department  to  brief  the  cadre  and  cadet  participants  on  the  purpose  of  the  investigation. 

Interested  parties  were  invited  to  sign  up  to  participate  in  the  investigation.  Volunteers  were 
randomly  assigned  to  either  the  standard  training  condition  or  the  extended  training  condition. 
Participants  were  then  assigned  time  slots  to  complete  their  assessment  and  training  obligations. 

Upon  arrival,  participants  were  greeted  and  seated  at  a  computer  terminal.  Participants  were 
then  given  a  brief  overview  of  the  project,  data  collection  and  use  of  the  data.  Then,  they  were 
given  a  brief  overview  of  the  project,  its  purpose,  a  description  of  the  data  collection  procedures, 
and  a  discussion  of  how  the  data  would  be  used.  After  preliminary  questions  were  answered, 
participants  completed  the  Skill  4  pretest.  After  completing  the  pretest,  participants  randomly 
assigned  to  the  extended  training  condition  completed  their  training,  followed  by  the  Skill  4 
posttest  and  a  post  training  questionnaire.  The  participants  who  had  been  assigned  to  the  standard 
training  version  followed  the  same  procedure,  except  they  completed  the  standard  training 
module.  After  completing  the  training  and  questionnaire,  both  training  groups  received  a  short 
email  inquiry  once  a  day  for  the  next  10  work  days,  asking  them  to  note  instances  of  how  the 
training  concepts  they  learned  applied  to  their  everyday  affairs.  The  participants’  responses  were 
submitted  via  email. 
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Results 


Participants’  posttest  scores  were  subjected  to  a  One-Way  Analysis  of  Covariance 
(ANCOVA)  that  examined  differences  in  post-test  scores  between  the  extended  and  standard 
training  groups,  using  participants’  pretest  scores  as  a  covariate  to  control  for  baseline  skill. 
There  was  no  significant  difference  found  between  the  post  test  performances  of  the  two  groups 
F(l,  18)  =  000 ,p>  .05.  The  pretest  and  posttest  means  (unadjusted  and  adjusted  for  pretest)  are 
listed  in  Table  7.  The  two  groups’  averages  were  almost  identical  indicating  that  the  extended 
version  did  not  produce  a  learning  advantage  over  the  2-hour  version  as  measure  by  Skill  4 
posttest  performance. 

Table  7.  Pretest  and  Posttest  Means  and  Standard  Deviations  by  Group  for  Investigation  2 


Group 

Pretest  Mean  (SD) 

Posttest  Mean  (SD) 

Posttest  Mean  (SD) 

(unadjusted) 

(adjusted) 

Extended  (N=8) 

120.2  (12.3) 

174.5  (34.6) 

174.4  (34.6) 

Standard  (N=13) 

120.2(10.3) 

174.5  (29.1) 

174.5  (29.1) 

Participants’  subjective  ratings  of  the  overall  training  were  analyzed  by  using  an  ANOVA 
comparing  the  average  rating  of  the  extended  version  to  the  average  rating  of  the  standard 
version.  The  ANOVA  showed  no  significant  difference  in  the  average  ratings  of  the  two  groups 
(M  =  1.84  for  standard  version;  M  =  1.75  for  extended  version.  F(l,  18)  =  .168 ,p  >  .05.  All  of 
the  participants,  except  one  in  the  2-hour  version  (who  rated  the  training  as  3-neutral),  rated  the 
training  as  either  very  good  (1)  or  good  (2),  reflecting  positively  on  the  training. 

Discussion 

The  results  suggest  that  the  standard  training  was  as  effective  as  the  extended  training  in 
teaching  the  skill  of  generalizing  from  specific  instances  to  broader  classes.  Given  the  limited 
time  often  available  for  extra  training,  these  results  suggest  that  the  standard  version  is  sufficient 
to  produce  a  learning  effect.  The  extended  version,  then,  could  be  used  if  a  student  wanted 
greater  explanation  or  needed  some  question  resolved.  The  extended  version  could  serve  as 
supplemental  material  which  could  be  made  available  for  those  trainees  who  desire  additional 
training  or  used  by  instructors  for  homework  or  for  additional  instructional  material. 


31 


FUTURE  RESEARCH 


This  multi-year  research  program  brings  to  researchers  in  the  field  of  critical  thinking  two 
important  resources  that  were  previously  unavailable  to  researchers.  The  first  is  a  testable  model 
of  CT  that  can  guide  future  research  efforts  on  the  construct.  The  second  is  (CT)2,  which 
provides  an  off-the-shelf  training  package  as  a  vehicle  to  conduct  the  research.  There  are  many 
possibilities  for  future  CT  research.  Below,  we  discuss  several  interesting  potential  topics. 

One  of  the  requirements  for  (CT)-  was  that  the  training  maximally  accessible  by  Soldiers 
anywhere  in  the  world.  Hence,  early  in  its  design,  the  design  decision  was  made  that  it  would  be 
delivered  via  the  Internet.  (CT)  was  to  be  a  web-based  program  that  would  provide  distance 
training,  perhaps  stand  alone,  on  CT  skills  important  to  Army  concerns.  Informational  content  is 
truly  unlimited  with  web-based  delivery,  and  computers  possess  an  exceptional  capability  to 
create  opportunities  for  practice.  However,  distance  training,  and  its  web-based  instantiation, 
impose  constraints  on  the  content  and  training  experiences  that  can  be  delivered  to  students.  We 
found  that  the  greatest  limitation  of  current  web-based  training  is  that  it  provides  only  limited 
capability  to  deliver  feedback  to  students’  responses.  For  a  skill  like  critical  thinking,  limitation 
on  the  quality  and  amount  of  feedback  might  severely  hamper  learning.  While  several 
evaluations  have  shown  that  (CT)"  does  produce  learning  and  increases  CT  skill,  it  is  likely  that 
feedback  tailored  to  students’  thinking  would  be  more  effective.  The  classic  pedagogy  that 
appears  to  be  maximally  effective  at  generating  clear  thinking  is  the  Socratic  method,  in  which  a 
tutor  points  out  the  weaknesses  of  a  student’s  thinking  by  asking  repeated  and  pointed  questions. 
The  technology  to  emulate  the  Socratic  method  on  a  computer  is  available  in  the  form  of 
intelligent  tutors.  In  recent  years,  largely  due  to  research  efforts  funded  by  the  United  States 
military,  intelligent  tutoring  technology  has  reached  impressive  capabilities.  It  is  now 
theoretically  possible  to  create  computer-based  CT  training  that  might  meet  this  high  standard  of 
feedback.  Future  research  should  explore  the  possibility  of  creating  a  true  Socratic  training 
system  designed  to  increase  CT  skills. 

(CT)"  rests  on  the  assumption  that  increasing  self-awareness  of  one’s  own  CT  will 
recursively  improve  the  same.  Indeed,  the  most  widely  used  textbooks  on  CT  (e.g.,  Paul  &  Elder; 
2001)  solely  use  exercises  that  increase  self-awareness  of  one’s  thinking.  Yet,  the  effectiveness 
of  increasing  self-awareness  on  improving  thinking  skills  has  not  been  empirically  shown.  More 
research  is  needed  to  further  investigate  the  relationship  between  meta-cognition  concerning  CT 
and  the  effective  use  of  CT. 

(CT)"  was  developed  to  improve  CT  in  individuals.  Yet,  an  emergent  property  of  team 
behavior  may  be  its  composite  ability  to  critically  think.  It  is  not  clear  whether  the  most  effective 
teams  are  composed  of  members  who  are  all  good  critical  thinkers,  or  have  only  one  or  two 
members  with  this  skill.  The  effect  of  the  distribution  of  CT  skills  across  a  team  likely  interacts 
with  communication  capability,  level  of  cooperation  and  other  team  characteristics.  Team 
composition  and  the  distribution  of  critical  thinking  skills  across  the  team  is  a  new,  potentially 
important,  area  of  study  that  should  be  pursued.  Further,  since  critical  thinking  in  the  Army  is 
most  often  conducted  as  a  team  activity  and  in  team  settings,  the  examination  of  the  process  of 
team  critical  thinking  and  methods  to  train  team  critical  thinking  would  be  relevant  and  valuable 
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research.  Preliminary  work  in  this  area  has  been  conducted  by  Cohen  and  his  colleagues  (Cohen, 
et  al.,  2006). 

The  relationship  between  intelligence,  ethics  and  CT  has  been  pursued  since  Watson  and 
Glaser  (1980)  first  conceptualized  CT  in  the  early  1940s.  Having  progressive  political  attitudes, 
these  early  researchers  hypothesized  that  liberal  viewpoints  were  related  to  high  levels  of  CT. 
Since  then,  others  with  conservative  political  perspectives  have  also  maintained  that 
conservatism  is  an  indicator  of  CT.  While  the  bias  is  obvious  in  these  positions,  the  fact  remains 
that  we  know  little  about  the  relationships  among  intelligence,  ethics,  and  CT.  We  can  create 
good  theories  of  CT  only  by  knowing,  via  empirical  demonstration,  what  it  is  not.  The  construct 
validity  of  CT  should  be  pursued  in  future  research,  or  the  field  will  continue  to  follow  the 
fragmented  path  that  has  been  its  course  since  the  1940s. 
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APPENDIX  A:  MODEL  OF  CRITICAL  THINKING 


Critical  thinking  was  first  investigated  in  the  early  1940’s  by  two  psychologists,  Goodwin 
Watson  and  Edward  Glaser.  Watson  and  Glaser  also  developed  the  first  test  of  CT,  the  Watson- 
Glaser  Critical  Thinking  Appraisal  (WGCTA)  (Watson  &  Glaser,  1980),  which  is  still  widely 
used.  Since  then,  almost  all  of  the  thinking  and  theoretical  development  of  CT  has  been 
conducted  by  educators  and  philosophers.  It  is  unclear  why  psychologists  have  played  a  small 
role  in  this  work;  however,  see  Halpem  (1996)  and  Baron  &  Sternberg  (1986)  for  notable 
exceptions.  Because  most  of  the  work  on  CT  has  been  conducted  by  educators  and  philosophers, 
the  construct  has  not  endured  the  kind  of  empirical  inspection  typically  bestowed  upon 
constructs  developed  by  psychologists.  Its  relationship  to  other,  well-established  psychological 
constructs  such  as  IQ,  working  memory,  and  reasoning,  for  example,  has  rarely  been  studied.  It 
is  the  authors’  admittedly  subjective  opinion  that  the  lack  of  empirical  investigation  of  CT  and  its 
relationship  to  other  individual  difference  dimensions  has  produced  a  fractionated  view  of  the 
construct.  Without  the  grounding  of  data,  theorists  have  been  free  to  postulate  divergent  concepts 
of  CT.  An  effort  in  philosophy  to  reach  a  consensus  definition  of  CT  in  1993  had  little  effect  on 
unifying  the  field  (American  Philosophical  Association,  1990). 

To  fill  this  gap,  Fischer,  Spiker,  and  Riedel  (2008a)  developed  a  model1  of  CT  that  is 
sufficiently  specified  to  permit  empirical  testing  and  falsification.  The  model  identifies  CT’s  role 
within  the  related  fields  of  reasoning  and  judgment,  which  have  been  empirically  studied  since 
the  1950s  and  are  better  understood.  The  model  incorporates  many  ideas  about  CT  offered  by 
leading  thinkers  (e.g.,  Paul  &  Elder,  2001)  in  philosophy  and  education.  It  also  embodies  many 
of  the  relevant  variables  discussed  in  the  CT  literature  (e.g.,  predisposing  attitudes,  experience, 
knowledge,  and  skills)  and  specifies  the  relationships  among  them.  The  model  can,  and  has  been, 
used  to  make  testable  predictions  about  the  factors  that  influence  CT  and  about  CT’s  psycho¬ 
logical  consequences  (Fischer,  et  ah,  2008b).  the  model  also  offers  practical  guidance  to  the 
development  of  CT  training.  In  this  Appendix  we  briefly  review  current  literature  about 
reasoning  and  judgment,  on  which  the  model  is  based,  and  describe  the  model’s  main  features. 

Dual  System  Theory  of  Reasoning  and  Judgment 

Prior  to  the  1950’s,  the  dominant  theories  held  that  people  made  judgments  by  calculating 
(1)  the  probability  and  (2)  the  utility  of  competing  options  .  Although  this  rational  choice  model 
took  on  a  variety  of  forms,  all  versions  posited  a  rational  actor  who  made  calculations  of 
probability  and/or  utility,  and  selected  the  option  that  had  the  highest  value.  In  the  1950’s, 
however,  researchers  began  to  notice  that  the  model  failed  to  predict  actual  behavior  (Meehl, 
1954;  Simon,  1957).  Evidence  that  falsified  the  rational  choice  theory  accumulated  over  the 
following  decade. 

In  the  early  1970s,  an  alternative  theory  proposed  that  people  use  heuristics,  as  opposed  to 
the  rational  weighing  of  relevant  factors,  to  make  judgments.  The  “new”  theory  was,  and 
continues  to  be,  supported  by  empirical  investigation.  The  heuristic  theory  states  that  many 
judgments  are  based  on  intuition  or  rules  of  thumb.  It  does  not  propose  that  all  judgments  are 
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made  intuitively,  just  that  there  is  a  tendency  to  use  such  processes  to  make  many  judgments. 

The  most  recent  versions  of  heuristic  theory,  in  fact,  propose  that  two  cognitive  systems  are  used 
to  make  judgments  (Kahneman,  2003).  The  first  system,  intuition,  is  a  quick,  automatic,  implicit 
process  that  uses  associational  strengths  to  arrive  at  solutions.  The  other,  reasoning,  is  effortful, 
conscious,  and  deliberately  controlled. 

Since  the  1970’s,  multiple  and  similar  two-process  theories  have  been  proposed  to  explain 
judgment.  To  accommodate  the  multiple  theories,  many  researchers  now  refer  to  the  implicit 
associational  type  of  process  as  System  1,  and  the  conscious  deliberate  process,  as  System  2.  The 
following  example,  taken  from  Kahneman  and  Frederick  (2002),  shows  how  these  two  processes 
may  lead  to  different  judgments. 

Suppose  a  bat  and  a  ball  cost  $1.10  in  total.  The  bat  costs  $1  more  than  the  ball.  How 

much  does  the  ball  cost? 

Most  people’s  immediate  judgment  is  that  the  ball  costs  10  cents.  This  is  a  response  derived 
from  intuition  or  System  1,  which  again,  is  quick,  automatic,  and  relies  on  associations.  The 
strong  mathematical  association  between  $1.10,  $1,  and  10  cents  leads  to  this  quick,  but  wrong, 
judgment.  The  ball  can’t  cost  10  cents  because  then  the  bat  would  have  to  be  $1,  which  would 
make  it  only  90  cents  more  than  the  ball.  The  more  effortful  deliberately  controlled  reasoning,  or 
System  2,  process  usually  produces  a  different,  and  correct,  answer.  When  people  spend  the  time 
and  effort  to  think  about  the  problem,  they  usually  realize  the  ball  must  cost  5  cents  and  the  bat 
must  cost  $1.05.  Hence,  in  this  example,  the  two  systems  produce  different  judgments.  It  would 
be  a  mistake  to  conclude  that  System  1  always  produces  different  judgments  than  System  2, 
however.  Nor  does  System  1  always  produce  an  incorrect  answer,  nor  one  that  is  poorer  than  one 
produced  by  System  2. 

In  fact,  researchers  have  shown  that  expert  performance  in  any  field,  which  is  commonly  the 
gold  standard,  is  often  driven  by  intuition  derived  from  extensive  experience  (e.g.,  Klein,  1999). 
That  said,  expert  performance  is  not  without  fault,  and  studies  have  shown  that  even  experts 
make  errors  in  judgment  when  well-learned  associations  lead  them  astray.  The  associational 
processes  used  in  System  1  that  make  expert  perfonnance  so  quick  and  powerful  are  the  same 
processes  that  are  responsible  for  systematic  errors  that  experts  sometimes  make.  Additional 
weaknesses  of  System  1  are  that  it  depends  on  the  quality  and  amount  of  experience  an  individ¬ 
ual  possesses,  and  it  can’t  be  used  effectively  in  novel  situations.  System  2  reasoning  also  has  its 
strengths  and  weaknesses.  While  it  is  highly  useful  in  novel  situations  and  problems,  it  is  also 
slow  and  effortful.  It  usually  cannot  be  utilized  concurrently  with  other  tasks  and,  like  System  1, 
it  can  also  produce  wrong  judgments. 

Most  researchers  posit  that  judgments  can  be  made  using  either  system.  Some  models  posit 
that  System  1  is  the  preferred  system  because  it  demands  fewer  resources  and  is  less  effortful  to 
apply.  Most  recent  theories,  however,  believe  that  Systems  1  and  2  run  in  parallel  and  work 
together,  capitalizing  on  each  other’s  strengths  and  compensating  for  weaknesses.  For  example, 
many  researchers  believe  that  one  function  of  the  controlled  deliberate  process  is  to  monitor  the 
products  of  the  automatic  process.  System  2  is  thought  to  endorse,  make  adjustments  to,  correct, 
or  block  the  judgment  of  System  1 .  However,  if  no  intuitive  response  is  accessible,  System  2 
may  be  the  primary  processing  system  used  to  arrive  at  judgment.  Sloman  (2002)  states  that  the 
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systems  work  hand  in  hand  as  “two  experts  who  are  working  cooperatively  to  compute  sensible 
answers.” 


The  similarities  between  descriptions  of  CT  and  System  2  are  striking.  The  words  “effortful, 
controlled,  deliberate,  purposeful,  and  conscious”  are  frequently  used  to  describe  both.  In  the 
next  section  we  will  see  that  the  Fischer,  et  al.’s  (2008a)  model  proposes  that  CT’s  primary 
function  is  to  monitor  and  control  the  judgments  produced  by  System  1  association  reasoning.  It 
further  proposes  that  System  2  is  the  engine  that  powers  CT  skills.  It  follows  that  errors  in 
reasoning  can  occur  when  CT  has  failed  to  serve  its  monitoring  and  correcting  function. 

Overview  of  CT  Model 

System  1  and  2  Engines.  As  shown  in  Figure  1,  the  CT  model  assumes  that  CT  skills  are 
executed  by  System  2. 2  CT  skills  serve  to  monitor,  evaluate,  and  control  the  judgments  produced 
by  the  System  1  associational  process.  Hence,  Figure  1  shows  that  System  1  solutions  provide 
input  to  CT  skills.  The  two  processes  are  thought  to  run  in  parallel  and  interact  to  produce 
judgments.  Because  System  1  is  truly  an  automatic  and  uncontrolled  process  it  cannot  be 
initiated  or  stopped.  For  this  reason,  CT  monitors  only  the  products,  and  not  the  process,  of 
System  1.  Because  System  1  is  quick,  it  often  comes  to  judgment  before  System  2.  However,  CT 
skills  executed  by  System  2  may  override,  or  confirm,  that  judgment.  Therefore,  System  2  has 
the  potential  for  controlling  judgment,  although  it  may  not  always  utilize  that  potential. 


CT  skills  can  provide  thorough  examination  of  the  problem  at  hand.  While  System  1 
typically  derives  only  one  solution  (Klein,  1999),  CT  skills  can  provide  multiple  potential 
solutions.  System  1  works  to  narrow  possible  action  paths,  which  is  often  highly  effective  when 
the  task  must  be  accomplished  quickly  and  when  the  problem  space  is  limited.  However,  when 
the  problem  space  is  novel  or  complex  or  when  solutions  must  be  innovative,  CT  skills  are  more 


Initial  Input 


Processing 

Engines 


Figure  1,  Model  of  critical  thinking  (Fischer,  et  al.,  2008b). 


We  do  not  claim  that  all  processing  executed  in  System  2  involves  CT  skills,  but  that  all  critical  thinking  requires 
the  kind  of  analytic,  rational  processing  that  only  System  2  can  provide.  For  example,  System  2  may  drive  other 
deliberate  processes  such  as  controlled  visualization.  However,  controlled  visualization  is  not  considered  a  CT  skill. 
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powerful.  CT  skills  have  the  meta-cognitive  capability  to  monitor  the  progress  of  their  own 
processing.  In  this  sense,  CT  is  recursive,  as  represented  by  the  arrows  leading  out  and  back  into 
the  “System  2:  CT  Skills”  processor  in  Figure  1 . 

Figure  1  also  shows  how  the  processing  engines  of  CT  and  System  1  interact  with 
environmental  and  individual  factors.  Both  systems  receive  initial  input  from  the  environment  in 
the  form  of  information  about  a  situation  or  problem  that  requires  judgment.  Part  of  that  input  is 
a  meta-task  that  defines  the  general  purpose  of  judgment.  The  other  part  of  the  input  is 
information  about  the  situation.  System  1  immediately  and  automatically  begins  processing  of 
the  input  by  searching  through  its  associational  network  for  potential  solutions  that  will  satisfy 
the  purpose.  CT  skills,  motored  by  the  System  2  processing  engine,  receive  the  same  input, 
filtered  through  predisposing  individual  difference  factors,  which  are  discussed  in  greater  detail 
below.  CT  skills  may,  or  may  not,  also  begin  processing  at  that  time.  If  CT  skills  are  engaged, 
they  will  begin  to  evaluate  solutions  offered  by  System  1  or  they  will  apply  deliberate  reasoning 
to  the  problem. 

Whether  or  not  CT  skills  are  utilized  depends  on  a  variety  of  factors,  including  individual 
predisposition  and  situational  variables.  The  sum  value  of  these  factors  provide  the  impetus  to 
engage  in  effortful  CT  skills,  but  that  motivation  must  exceed  some  threshold  value  for  CT  to  be 
initiated.  Let’s  examine  each  component  of  the  model  in  more  detail. 

The  Context:  Situation  and  Meta-Task.  As  noted  above,  the  opportunities  for  judgment  are 
set  in  motion  by  contextual  factors,  which  include  (1)  the  situation  and  (2)  purpose  (meta-tasks). 
While  the  automatic  System  1  will  engage  in  all  conditions,  two  characteristics  of  the  situation 
must  be  present  to  elicit  CT  activity.  First,  stimulus  material  must  contain  substantive  informa¬ 
tion.  Second,  there  must  be  sufficient  time  available  to  engage  in  an  appropriate  CT  skill.  Other 
characteristics  of  the  situation  make  it  more  likely  that  CT  will  be  initiated.  For  example,  the 
presence  of  conflicting  information,  disordered  or  unorganized  material,  uncertain  infonnation, 
and  complex  material  all  make  it  more  likely  that  CT  will  be  engaged. 

CT  is  not  an  end  in  itself,  but  serves  other  objectives  specified  by  purpose  (meta-tasks).  The 
purpose  also  dictates  the  specific  response  that  will  be  required  to  successfully  end  the  execution 
of  a  CT  skill.  For  example,  the  situation  may  include  a  meta-task  to  understand ',  make  an 
evaluation,  make  a  decision,  or  solve  a  complex  problem.  Whether  final  judgment  is  based  on 
CT  or  System  1  processing,  System  2  detennines  when  the  requirements  of  the  purpose  have 
been  met.  Hence,  successful  completion  of  the  meta-tasks  as  detennined  by  System  2  can  also 
provide  input  that  terminates  a  CT  episode,  as  depicted  in  the  bidirectional  arrow  in  Figure  1. 

Predisposing  Factors.  Predisposing  factors  influence  the  likelihood  of  a  person  using,  or 
persisting  in  using,  a  CT  skill.  Like  features  of  the  situation,  they  serve  as  input  conditions  to  CT 
skill,  and  as  a  filter  through  which  the  situation  and  purpose  are  evaluated.  Some  may  be  key 
factors  that  strongly  affect  an  individual’s  use  of  a  CT  skill.  Other  factors  may  have  a  weaker 
relationship  to  CT,  perhaps  increasing  the  likelihood  of  engaging  in  CT  by  a  marginal  amount.  In 
summary,  predispositions  are  measurable  ways  in  which  people  differ,  whether  fixed  or  modifi¬ 
able,  that  influence  the  use  or  persistence  of  use  of  a  CT  skill. 
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Moderating  Variables,  Education,  and  experience  Moderating  variables  are  individual 
variables  that  influence  how,  and  how  well,  CT  skills  are  perfonned.  For  example,  domain 
expertise,  recent  experience,  and  education  influence  the  quality  of  the  reasoning  produced  by 
CT.  They  do  not,  however,  influence  whether  one  executes  a  CT  skill,  as  do  predisposing  factors. 

Negative  Experiential  Consequences.  There  is  a  general  consensus  in  the  literature  that 
individuals  are  reluctant  to  engage  in  CT.  This  is  based  on  widespread  observation  of  incoherent 
reasoning,  nonsensical  beliefs,  lack  of  respect  for  evidence,  poor  CT  test  scores,  and  unsupported 
decision-making  in  various  American  populations.  Indeed,  much  of  the  CT  literature  is  devoted 
to  a  movement  to  increase  the  application  of  CT  in  various  populations.  One  of  the  central  topics 
has  been  the  question  of  why  the  public  seems  disinclined  to  use  CT.  Some  theorists  posit  that 
individual  characteristics,  such  as  intellectual  laziness,  arrogance  and  cowardice  (which  are 
represented  in  the  model  as  predisposing  individual  differences),  are  the  reasons  why  CT  is 
avoided.  The  model  of  CT  discussed  here,  however,  also  posits  that  negative  affective 
consequences  associated  with  the  application  of  CT  are  inhibitory  sources. 

The  model  posits  that  individuals  who  engage  in  CT  for  any  substantive  length  of  time  are 
likely  to  experience  negative  affective  reactions.  For  example,  CT  can  produce  mental  fatigue, 
increased  effort,  increased  anxiety,  cognitive  dissonance,  and  decreased  self-esteem.  Negative 
affect  experienced  during  a  CT  episode  might  be  countered  by  positive  affect  that  is  the  result  of 
a  positive  outcome  (e.g.,  solving  a  difficult  problem)  that,  in  turn,  is  a  direct  result  of  CT.  There¬ 
fore,  the  application  of  CT  may  be  positively  rewarded  and  hence,  increases  in  its  use  may  be 
realized.  Some  individuals,  then,  may  not  experience  negative  affect  associated  with  CT.  But  at 
the  very  least,  by  definition,  CT  requires  more  effort  than  System  1  processing,  and  is  therefore  a 
less  desirable  means  to  achieve  judgment  in  that  limited  sense. 

Figure  1  shows  that  affective  experiential  consequences  serve  as  both  a  byproduct  of  CT  and 
as  input  to  the  decision  to  maintain  a  CT  episode,  as  depicted  by  the  bidirectional  arrow  in  Figure 
1 .  When  the  affective  consequences  of  applying  a  CT  skill  become  too  negative,  the  motivation 
to  maintain  the  episode  is  decreased.  If  the  negative  consequences  are  sufficiently  strong,  they 
may  result  in  a  cessation  of  the  episode.  It  is  important  to  note  that  research  conducted  in  our  lab 
has  indicated  that  engagement  in  CT  does  not  always  produce  corresponding  negative  affect 
(Fischer  &  Spiker,  2004a). 

System  1  and  2  Products.  As  shown  in  Figure  1,  Systems  1  and  2  have  the  capacity  to 
generate  products  in  the  form  of  judgments.  The  double  arrow  leading  to  and  away  from  System 
2  solutions  indicates  that  this  deliberate  process  is  capable  of  evaluating  its  own  products.  In 
contrast,  the  single  arrow  leading  from  System  1  to  its  judgment  product  shows  that  this 
associational  process  does  not  have  that  same  meta-cognitive  capability.  Note  also  that  the 
System  1  judgments  also  serve  as  input  to  System  2. 

The  Quality  of  Judgment.  The  task  posed  by  a  particular  situation  should  not  be  confused 
with  the  system  that  is  used  to  solve  it.  For  example,  one  may  have  the  task  of  understanding  a 
commander’s  intent  statement  that  could  be  achieved  using  associational  processes  of  System  1 
or  controlled  CT  skills  powered  by  System  2.  Therefore,  an  individual  who  is  trying  to 
understand  an  intent  statement  may  or  may  not  be  using  a  CT  skill  to  do  so.  Even  more 
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important,  the  application  of  CT  skills  driven  by  System  2  does  not  always  produce  the  best 
solution  to  a  task.  It  would  be  a  mistake  to  encourage  consistent  use  of  CT  because  that  strategy 
would  deny  the  power  and  effectiveness  of  System  1 .  Similarly,  it  is  not  advisable  to  only 
develop  associational  processes  because  controlled  deliberate  reasoning  can  (1)  produce  superior 
solutions,  and  (2)  provide  necessary  checks  on  the  products  of  System  1 .  Moreover,  the  issue  of 
which  system  is  most  effective  is  practically  irrelevant  because  most  theorists  believe  that  both 
are  almost  always  used  in  conjunction  to  produce  a  solution.  Hence,  the  real  issue  that  deter¬ 
mines  the  quality  of  a  solution  is  probably  how  well  the  two  systems  interact. 

The  quality  of  a  solution  produced  by  the  application  of  a  CT  skill  may  also  be  affected  by 
how  well  the  skill  is  executed.  Decrements  in  performance  may  be  produced  by  failing  to  apply  a 
component  of  the  CT  skill  (e.g.,  failing  to  clarify  ambiguous  infonnation  in  a  message  or  failing 
to  consider  alternative  explanations  for  a  pattern  of  data),  failing  to  accurately  perform  a 
component  of  the  skill,  or  by  lacking  sufficient  knowledge  that  can  be  processed  by  the  CT  skill. 
Therefore,  one  could  apply  a  CT  skill  and  still  produce  inferior  solutions  to  a  task.  Moreover,  it 
is  not  possible  to  determine  whether  System  1  or  System  2  was  applied  to  derive  a  solution  based 
on  the  solution  alone.  The  quality  of  a  solution  may  also  be  affected  by  moderating  variables 
such  as  educational  level  and  experience.  These  issues  are  important  to  the  design  of  training  that 
seeks  to  improve  CT  skills.  The  model  of  CT  presented  here  was  used  to  identify  a  pedagogical 
approach  and  principles  of  training  CT.  This  approach  was  then  used  to  design  (CT)2 
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APPENDIX  B:  FUNCTIONAL  SPECIFICATIONS 


CT  SKILL  1:  FRAME  THE  MESSAGE 
WHAT  THIS  SKILL  IS  ABOUT 
Skill  Definition 

This  skill  is  the  ability  to  identify  the  essential  elements  of  a  message,  understand  their  relationships,  and  describe  a 
high  fidelity  representation  of  the  message. 

Description  of  Skill 

First,  let's  consider  what  is  meant  by  a  "message".  In  this  training,  "message"  refers  to  any  stimulus  material  that  is 
relevant  to  a  task  that  must  be  performed  by  an  Army  officer.  It  may  be  written  on  a  computer  screen,  a  piece  of 
paper,  or  in  the  sand.  It  might  also  be  an  audio  transmission,  recorded  or  live.  A  message  does  not  necessary  use 
language  to  communicate;  it  could  use  graphics.  Thus,  a  message  might  be  a  commander's  intent  statement,  a 
situation  map,  a  decision  support  template,  a  radio  transmission,  and  so  forth. 

Frame  the  Message  is  a  core  skill  that  critical  thinkers  use  to: 

•  identify  key  information  elements  in  a  message, 

•  discern  the  meaning  of  all  message  components,  and 

•  identify  what  is  known  and  unknown  given  the  message,  and 

•  identify  inferences  that  can  be  easily  derived  from  the  message  vs.  inferences  that  can  only  be  derived  if 
certain  assumptions  (based  on  uncertain  information)  are  applied 

Next,  let’s  consider  what  is  meant  by  “frame”.  The  frame  is  not  an  exact  replica  of  the  message.  Rather  it  is  a 
structure  that  can  hold  all  of  the  message’s  components.  It  is  an  abstract  representation  of  the  message.  Just  as  a 
carpenter  builds  a  house  by  first  creating  the  framing  or  supporting  structure  -  the  foundation,  the  two  by  four's,  the 
rebar,  and  so  forth  -  so  too,  the  critical  thinker  begins  his/her  analysis  of  a  message  by  framing  its  content.  The 
frame  of  a  house  provides  support  for  the  details  it  will  later  enjoy,  such  as  windows,  doors,  a  roof,  panels,  etc.  In  a 
similar  fashion,  the  frame  of  a  message  supports  the  content  the  message  contains.  But  just  as  the  frame  of  a  house 
determines  many  of  the  structure’s  features  (e.g.,  size,  shape,  facilities  and  functions),  the  frame  a  person  adopts  for 
a  message  will  determine  how  that  message  is  understood. 

The  frame  of  a  message  can  be  regarded  as  the  foundational  structure  of  the  information.  As  a  foundation,  the  frame 
will: 


•  indicate  the  information  boundaries,  or  topics,  contained  within  the  message 

•  represent  information  given  in  the  message 

•  highlight  the  weak  or  questionable  items 

•  denote  how  message  content  elements  are  related 

•  indicate  content  that  may  be  missing  from  the  message 

•  indicate  content  that  may  be  incorrect  in  the  message 

A  Frame  is  a  mental  structure  that  helps  us  interpret  information.  Frames  provide  the  scaffolding  on  which 
information  can  "hang".  Frames  have  powerful  effects  on  peoples’  understanding  of  situations,  information,  and  the 
world  around. 

Battle  Command  Tasks  Associated  with  Skill 

The  target  battle  command  task  for  which  this  skill  is  required  is  “clarify  the  intent  of  the  commanders  1  and  2  levels 
up”.  However,  this  skill  also  facilitates  many  other  military  tasks  that  involve  interpreting  messages  including,  but 
not  limited  to: 
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1 .  Review  a  mission  statement 

2.  Determine  own  unit’s  area  of  responsibility 

3.  Determine  constraints/restraints  placed  on  mission  by  higher  command 

4.  Indeed,  this  skill  can  be  applied  to  any  battle  command  task  that  involves  the  reception  and  interpretation  of 
a  message,  whether  that  message  is  delivered  in  print,  graphic,  or  auditory  (e.g.,  radio  transmission)  form. 

Types  of  Errors 

A  variety  of  errors  are  common  when  executing  this  skill: 

1.  Select  wrong  frame  for  message 

2.  Fail  to  place  information  in  frame  slot 

3 .  Place  wrong  information  in  slot 

4.  Fail  to  place  ALL  information  in  slot 

5.  Fail  to  identify  or  recognize  weak  spots  in  message 

6.  Fail  to  resolve  weak  spot  in  message 

7.  Make  unjustified  inferences: 

8.  Substitute  what  a  message  should  say  for  what  it  does  say 

9.  Interpret  content  of  message  to  fit  preconceived  notions 

10.  Believe  that  own  inferences  are  given  as  evidence  in  message 

11.  Fail  to  recognize  probability  that  inferences  developed  in  framing  the  message  is  correct  or  incorrect 


Learning  Objectives  for  this  Skill 

This  training  module  will  focus  on  a  set  of  learning  objectives  that  constitute  aspects  or  components  of  this  CT  skill. 
Upon  successful  completion  of  the  training,  the  student  will  be  able  to: 

1 .  Identify  an  unstructured  message's  components 

2.  Understand  and  identify  six  different  reasons  why  a  message  may  contain  weak  spots 

3.  Distinguish  between  weak  messages  based  on  unclear  information  and  those  based  on  uncertain 
information 

4.  Understand  the  steps  involved  in  resolving  six  types  of  weak  spots 

5.  Distinguish  actual  evidence  from  inferred  assumptions 

6.  Gauge  the  strength  of  the  inference  made  in  framing  the  message 

Skill  Boundaries 

The  most  closely  related  skill  is  CTS  2,  Recognize  Gist  in  Material.  The  two  skills  differ,  however,  in  their  goals 
and  end  products.  The  goal  of  Frame  the  Message  is  to  thoroughly  examine  a  message  and  to  represent  each  of  its 
components,  whether  they  are  key  or  incidental.  In  contrast,  the  goal  of  Recognize  Gist  in  Material  is  to  extract  and 
represent  only  the  key  components  of  a  message.  The  ideal  end  product  of  Frame  the  Message  is  a  restatement  of 
the  message,  with  each  component  analyzed  and  understood  to  the  degree  possible.  The  ideal  end  product  of 
Recognize  Gist  in  Material  is  a  summarization  of  the  message  that  captures  its  most  important  meaning. 

CTS  1  is  also  related  to  other  CT  skills  and  often  forms  a  first  step  because  CT  is  often  based  on  messages  crafted  by 
other  individuals.  In  particular,  CTS  1  is  related  to  CTS  6  Challenge  One’s  Bias,  and  CTS  7  Examine  Other 
Peoples  ’  Perspectives. 

A  central  challenge  in  CTS  6  is  to  recognize  and  correct  confirmation  bias,  which  can  also  influence  how  one 
Frames  a  Message.  Confirmation  bias,  however,  can  be  applied  not  only  to  messages,  but  also  situations, 
explanations,  predictions,  etc.  CTS  1  is  limited  to  understanding  messages,  whether  they  are  in  text,  auditory,  or 
some  other  form. 

Examining  another  person’s  perspective  (CTS  7)  can  be  applied  to  framing  a  message  crafted  by  another  person. 
Both  skills  focus  on  other  people’s  thinking  or  products.  However,  CTS  7  requires  and  guides  extensive  inferencing, 
while  CTS  1  attempts  to  recognize  and  limit  inferences.  The  goal  of  CTS  1  is  to  stick  to  what  is  given  in  the 
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message,  and  recognize  information  that  is  not  given  that  might  be  related  to  inferences  that  might  be  made.  The 
goal  of  CTS  7  is  to  open  one’s  mind  to  another  person’s  goals,  and  to  NOT  limit  one’s  understanding  of  another 
person.  By  its  nature  CTS  7  demands  careful  inference.  CTS  1  demands  limitations  be  placed  on  inferences. 


Flow  of  Training 

1 .  The  training  will  begin  with  an  interactive  experience  that  shows  the  student  how  frames  are  important 
how  they  influence  thinking.  Students  will  be  exposed  to  messages  for  which  a  frame  cannot  be 
applied  without  additional  information.  The  message  will  be  uninterpretable  until  the  frame  is  given. 

2.  Then,  students  will  be  given  a  series  of  exercises  designed  to  promote  thorough  examination  of  every 
component  of  the  message.  The  exercises  will  ask  students  to  place  each  of  the  message  components 
into  a  category  of  agent,  object,  etc. 

3.  Going  deeper  into  analyzing  the  message,  students  will  next  be  given  a  set  of  exercises  designed  to 
sensitive  them  to  weak  spots  (messages  components  whose  meaning  cannot  be  easily  accessed  from 
the  message  alone)  in  a  message.  The  exercises  will  ask  students  to  identify  if  weak  spots  are  present 
and  then  decide  why  they  are  weak  spots. 

4.  The  weak  spot  exercises  will  be  followed  with  several  exercises  designed  to  train  students  in  the 
resolution  of  weak  spots. 

5.  Then,  students  will  be  asked  to  examine  inferences  that  might  be  made  about  weak  spots. 

6.  A  post  test  will  conclude  the  training. 

Training  Exercises 

The  training  will  include  the  following  exercises,  broadly  described. 

Training  Exercise  Set  1:  Agent— Object 
Training  Exercise  Set  2:  Weak  Spots 
Training  Exercise  Set  3:  Resolution  of  Weak  Spots 
Training  Exercise  Set  4:  Inference  Making 
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CT  SKILL  2:  RECOGNIZE  GIST  IN  MATERIAL 


WHAT  THIS  SKILL  IS  ABOUT 
Skill  Definition 

This  critical  thinking  skill  is  the  ability  to  sort  through  the  details  in  a  message  (written,  graphical,  visual,  auditory, 
and/or  tabular)  and  extract  the  gist  therein. 

Description  of  Skill 

This  skill  involves  the  ability  to  sort  through  the  details  of  a  message  and  distinguish  essential  from  non-essential 
information.  It  involves  the  ability  to  generate  a  clear,  concise  statement  that  summarizes  the  gist  or  main  point  of  a 
complex  message. 

Recognizing  and  extracting  the  gist  of  a  message  is  a  useful  skill  because: 

♦  the  gist  serves  as  a  convenient  shorthand  to  help  communicate 

♦  the  gist  is  more  easily  remembered 

♦  the  gist  helps  one  keep  focused  on  the  most  important  issues  at  hand 

Furthermore,  the  process  of  extracting  the  gist  is  a  useful  skill  because: 

♦  it  requires  deeper  processing  of  the  message 

♦  it  highlights  where  clarification  is  needed 

♦  it  helps  one  evaluate  how  parts  of  the  message  fit  together 

Extracting  the  gist  of  a  message  is  often  an  automatic  process;  people  rarely  remember  a  message  verbatim,  instead 
they  remember  the  parts  that  seem  relevant  to  them.  This  is  often  referred  to  as  the  “take  home  message.”  This 
process  generally  works  well  if  the  original  message  is  simple  and  the  main  points  are  stated  clearly  and 
unambiguously.  However,  messages  have  varying  degrees  of  complexity.  Some  are  created  for  a  wide  audience,  and 
it  is  up  to  the  recipients  to  extract  the  gist  that  is  most  relevant  for  their  purposes.  Other  messages  are  wordy  or 
poorly  written  and  the  recipient  must  sift  through  the  details  to  determine  what  is  and  isn’t  important. 

When  extracting  the  gist,  the  recipient  often  attempts  to  make  sense  of  the  message  by  dropping  things  that  are 
unfamiliar  and  by  using  their  prior  knowledge  to  fill  in  any  gaps.  This  is  related  to  Bartlett’s  schema  theory,  which 
examined  the  processes  people  engage  in  when  remembering  and  recounting  information.  Bartlett  found  that  when 
people  retold  stories,  the  stories  changed  in  systematic  ways.  Specifically,  details  such  as  names  and  places,  as  well 
as  themes  that  are  unfamiliar  to  the  reader  tend  to  be  dropped,  while  other  details,  especially  vivid  ones,  tend  to  be 
elaborated  upon.  Another  common  change  is  the  addition  of  explanations  or  rationalizations  that  are  consistent  with 
the  readers’  expectations,  and  which  help  them  make  sense  of  the  story. 

These  systematic  changes  are  normal  and  expected,  but  they  can  also  lead  to  misunderstandings  and  the  distortion  of 
messages,  particularly  when  the  gist  of  the  message  is  passed  on  from  person  to  person.  Other  common  problems 
that  people  have  when  extracting  the  gist  are  being  too  vague,  focusing  on  supporting  details  rather  than  on  the  main 
point,  focusing  on  irrelevant  details,  or  missing  the  main  point  altogether.  Thus,  an  important  part  of  extracting  the 
gist  of  a  message  involves  creating  a  deliberate  awareness  of  how  one  is  evaluating  what  is  and  is  not  important,  and 
requires  one  to  be  cognizant  of  the  ways  in  which  messages  can  be  altered  or  distorted. 

Subcomponents  of  the  skill  are: 

1)  considering  context  -  why  is  the  author  sending  the  message  and  what  is  the  recipient’s  role? 

2)  weighing  the  relative  relevance  of  specific  bits  of  information 

3)  combining  and  rephrasing  information  to  make  it  more  concise 

4)  generating  a  compact  gist  statement  that  is  relevant,  accurate,  and  meaningful. 

5)  reviewing  the  gist  statement  to  make  sure  that  anything  that  was  left  out  or  added  does  not  alter  the  intended 
meaning  of  the  message,  or  lead  to  possible  misinterpretations  later  on. 

Battle  Command  Task  Associated  with  Skill 
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The  target  battle  command  task  for  which  this  skill  is  required  is  restating  mission  objectives  provided  by  upper 
echelons  in  order  to  write  one’s  own  mission  statements.  However,  this  skill  also  facilitates: 


1.  Better  memory  for  and  focus  on  relevant  tasks 

2.  Better  communication 

3.  Opportunity  for  clarification  of  the  intended  purpose  of  the  message 

4.  Developing  solutions  and  making  decisions  about  what  is  relevant  to  one’s  unit 

5.  Prioritization 

Types  of  Errors 

A  variety  of  errors  are  common  when  executing  this  skill,  including: 

1.  Failure  to  distinguish  relevant  from  irrelevant  information 

2.  Failure  to  eliminate  less  important  details  from  the  central  meaning  of  message 

3.  Writing  or  presenting  overly  detailed  OPORDS  or  failure  to  provide  the  bottom  line  up  front  (BLUF) 

4.  Failure  to  accurately  extract  the  essence  of  a  mission 

5.  Inserting  unsupported  rationalizations  or  exaggerating  details 

6.  Leaving  out  potentially  important  information  merely  because  it  is  unfamiliar 

7.  Being  too  vague  in  recounting  the  gist 

Learning  Objectives  for  this  Skill 

This  training  module  will  focus  on  a  set  of  learning  objectives  that  constitute  aspects  or  components  of  this  CT  skill. 
These  are: 

1 .  Recognizing  good  and  poor  quality  gist  statements 

2.  Identifying  errors  in  extracting  the  gist 

3.  Understanding  the  importance  of  context 

4.  Applying  techniques  to  select  and  extract  central  themes  and  most  important  points  from  a  message 

5.  Combining  information  and  revising  to  make  it  more  concise 

6.  representing  the  essential  points  of  a  message  in  a  clear,  concise  manner 

7.  Checking  for  unsupported  inferences  and  distortions  of  the  original  message 

8.  recognizing  how  the  process  of  distilling  the  main  idea  encourages  deeper  processing  and  can  lead  to 
increased  ability  to  remember,  evaluate,  and  use  the  information 

Skill  Boundaries 

The  most  closely  related  skill  is  CTS  1,  Framing  the  Message.  CTS  1  develops  the  ability  to  parse  messages  and 
examine  specific  details  so  that  any  flaws  or  weak  spots  in  the  message  -  such  as  inconsistencies,  missing 
information,  ambiguous  information,  questionable  information  -  can  be  detected.  CTS  2,  in  contrast,  is  concerned 
with  filtering  out  extraneous  details  and  focusing  on  the  most  important  main  point  of  the  message. 

CTS  2  also  contains  elements  relating  to  CTS  6,  Challenge  One’s  Bias.  When  restating  a  message,  it  is  common  for 
personal  biases  to  influence  what  we  remember  and  what  we  believe  to  be  important;  thus,  we  may  ignore  details 
that  don’t  fit  our  pre-existing  ideas  of  what  the  message  should  say.  Likewise,  we  may  exaggerate  some  details  or 
provide  rationalizations  that  were  not  in  the  original  message  in  order  to  make  it  fit  with  our  expectations. 


Flow  of  Training 

1 .  The  training  will  begin  with  an  interactive  experience  that  shows  the  student  how  extracting  the  gist  of  a 
message  is  in  essential  to  communication  and  decision  making. 

2.  The  second  section  will  review  the  qualities  of  a  good  gist  statement,  and  common  errors  people  make  in 
extracting  the  gist 

3.  The  third  section  will  review  a  5-step  technique  to  systematically  and  accurately  extract  the  gist  from 
complex  messages.  This  section  will  walk  the  user  through  a  series  of  brief  exercises  designed  to  focus 
attention  on  the  meaning  and  purpose  of  each  step. 

4.  Then,  there  will  be  a  series  of  exercises  that  provide  an  opportunity  for  the  student  to  apply  the  technique 
they  just  learned  to  a  variety  of  military-themed  messages. 
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5. 


Finally,  we’ll  provide  exercises  on  applying  the  technique  to  non-text  messages,  such  as  graphics  and 
tables. 

6.  Training  will  end  with  summary  of  what  they  were  expected  to  get  out  of  the  module,  and  where  it  might 
be  applicable. 

7.  The  module  will  end  with  a  Post  test. 

Training  Exercises 

The  training  will  include  the  following  exercises,  broadly  described. 

1.  Beginning  scenario 

2.  Training  Set  1:  Recognizing  a  good  gist  statement 

3.  Training  Set  2:  Applying  a  5-step  technique  for  extracting  the  gist  from  complex  messages 

4.  Training  Set  3:  Applying  the  12-8-4  Technique  to  non-text  messages 
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CT  SKILL  3:  DEVELOP  AN  EXPLANATION  THAT  TIES  INFORMATION 
ELEMENTS  TOGETHER  IN  A  PLAUSIBLE  WAY 

WHAT  THIS  SKILL  IS  ABOUT 

Skill  Definition 

This  skill  is  the  sum  of  the  abilities  to: 

•  Arrange  facts  logically. 

•  Highlight  gaps  in  knowledge. 

•  Develop  an  explanation  or  multiple  explanations  of  the  event/phenomenon  in  question 

•  Evaluate  the  explanation(s)  for  plausibility. 

Description  of  Skill 

The  essence  of  the  skill  is  the  ability  to  look  at  evidence  from  multiple  perspectives,  assess  its  credibility  and 
meaning,  and  develop  one  or  more  explanations  that  account  for  the  evidence  and  make  sense  for  the  current 
mission  or  task.  The  product  of  applying  the  skill  is  generation  of  useful  evidence-based  explanations. 

Battle  Command  Task  Associated  with  Skill 

The  target  battle  command  task  for  this  skill  is  to  interpret  reports  of  recent  enemy  activities  in  the  area  of  interest  to 
estimate  enemy  intent  and  predict  enemy  actions. 

Types  of  Errors 

Errors  commonly  made  when  executing  this  skill  fall  into  one  of  two  general  categories,  prejudicial  treatment  of 
evidence  and  failure  to  look  broadly  enough  at  plausible  explanations. 

Some  typical  evidential  errors  are: 

•  Ignoring  evidence  that  doesn’t  fit  a  preconceived  story  or  preferred  conclusion 

•  Failure  to  identify  important  relationships  within  the  evidence 

•  Failure  to  distinguish  between  causal  and  associative  evidence 

•  Failure  to  incorporate  all  evidence  into  explanation 
Examples  of  explanatory  errors  include: 

•  Selecting  only  explanations  that  fit  a  preconceived  story 

•  Constructing  causal  chains  based  on  prejudice  or  preference,  without  evidential  justification 

•  Failure  to  build  more  than  one  explanation,  a  priori  selecting  the  preferred  one  or  the  worst-case 

•  Judging  explanations  based  on  preferred  outcome  instead  of  objective  standards 

Learning  Objectives  for  this  Skill 

The  terminal  learning  objective  for  this  skill  is  the  ability  to  see  multiple  possible  explanations  in  a  body  of  evidence 
and  select,  without  over-commitment,  those  that  best  fit  the  available  evidence. 

Upon  successful  completion  of  the  training,  the  student  will  be  able  to: 

1 .  quickly  generate  multiple  explanations 

2.  quickly  organize  evidence,  and  to  be  aware  of  gaps 

3.  see  relationships  among  elements  of  evidence,  both  reinforcing  and  disconfirming 

4.  connect  evidence  to  explanations  without  distortion 

5.  select  explanations  based  on  dispassionate  assessment  of  fit  with  evidence 

Skill  Boundaries 

Because  comprehension  of  the  situation  is  central  to  so  many  battle  command  tasks,  this  skill  overlaps  with  other 
skills.  Frame  the  Message,  Generalize,  Recognize  Your  Biases,  Decide  When  to  Seek  highly  related  to  Examining 
Other  People’s  Perspectives  as  they  will  inform  alternative  explanations. 

Flow  of  Training 

1 .  The  training  will  begin  with  exercises  that  give  students  the  opportunity  to  generate  multiple  explanations. 
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2.  Then,  students  will  be  taught  principles  of  gathering  and  organizing  evidence. 

3.  Going  deeper  into  the  validity  and  quality  of  the  evidence  students  will  be  given  exercises  in  critically 
examining  evidence. 

4.  Students  will  then  be  taught  how  to  select  a  working  explanation. 

5.  A  post  test  will  conclude  the  training. 

Training  Exercises 

The  training  will  include  the  following  exercises,  broadly  described. 

Training  Exercise  Set  1:  Generate  at  least  one  explanation  from  these  perspectives:  first  thought,  describe  the  facts, 
describe  the  motivation,  describe  from  “their”  perspective,  describe  from  my  perspective. 

Training  Exercise  Set  2:  Create  a  table  of  evidence  you  and  don’t  have. 

Training  Exercise  Set  3:  Check  evidence  for  internal  contradiction. 

Training  Exercise  Set  4:  Compare  explanations  using  method  of  triangulation. 
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CT  SKILL  4:  GENERALIZE  FROM  SPECIFIC  INSTANCES  TO  BROADER  CLASSES 


WHAT  THIS  SKILL  IS  ABOUT 
Skill  Definition 

This  skill  is  the  ability  to  recognize  and  then  classify  specific  observations,  facts,  incidents,  or  events  into  a  broader 
class.  This  skill  is  successfully  executed  when  there  is:  an  accurate  observation  and  description  of  the  individual 
instance,  identification  of  multiple  classes  to  which  the  instance  might  be  generalized,  an  assessment  of  how  well  the 
instance  fits  the  different  classes,  and  an  indication  of  how  that  instance  might  change  over  time  to  fit  another  class. 

Description  of  Skill 

The  essence  of  this  skill  is  the  ability  to  make  three  types  of  generalization  rapidly  and  accurately.  The  three  types  of 
generalization  are: 

•  Categorization — putting  an  instance  into  a  larger  bin.  Is  a  fleeing  vehicle  an  enemy  BMP? 

•  Trending — determining  whether  something  is  growing,  shrinking,  or  expanding.  Is  a  crowd  of  potentially 
hostile  local  citizens  growing  or  staying  steady? 

•  Reconstitution — determining  whether  a  partially  masked  or  degraded  observation  is  an  object  of 
importance.  Is  a  small  roadside  object  with  protruding  wires  an  IED? 

Battle  Command  Tasks  Associated  with  Skill 

The  target  battle  command  task  for  use  with  this  skill  is  interpreting  reports  of  enemy  disposition  based  on  the 
availability  of  only  partial  information.  Other  battle  command  tasks  that  are  aided  by  this  skill  include: 

•  If  2  local  citizens  are  seen  on  the  street,  does  that  mean  a  (potentially  hostile)  crowd  is  forming? 

•  If  a  roadside  mine  is  detected,  does  that  mean  a  mine  field  is  nearby? 

•  If  we  see  a  partially  cloaked  individual  with  some  characteristics  of  a  terrorist,  does  that  mean  he  is  one? 

•  If  we  see  some  elements  of  a  motorized  rifle  regiment,  does  that  mean  the  rest  of  the  unit  is  nearby? 

•  If  we  see  a  partial  glimpse  of  a  low,  fast  flyer,  is  it  an  enemy  or  neutral  aircraft? 

Types  of  Errors 

A  variety  of  errors  are  common  when  executing  this  skill: 

1 .  fail  to  accurately  induce  patterns  of  overall  movement  based  on  report  instances 

2.  tendency  to  disregard  reports  that  do  not  match  expectations 

3.  tendency  to  inflate  information  on  reports 

Learning  Objectives  for  this  Skill 

The  terminal  training  objective  is  the  ability  to  make  rapid,  accurate  categorizations,  trend  assessments,  and 
generalizations  based  on  complete  descriptions  of  observed  instances. 

There  are  four  enabling  objectives: 

1 .  Accurately  describe  observed  instances  without  missing  any  important  features  in  an  unbiased  manner. 

2.  Comprehensively  identifying  alternative  classes  in  which  the  instance  might  fit  by  considering  counter¬ 
evidence  that  would  indicate  a  non-fit. 

3.  Evaluating  how  well  the  observed  instance  fits  into  alternative  classes  by  considering  the  most  relevant 
features. 

4.  Evaluating  how  the  categorization  decision  might  change  over  time  by  considering  the  relative  permanence 
of  the  distinguishing  features. 


Skill  Boundaries 

The  most  closely  related  skill  is  CT  Skill  3,  “Generate  a  Plausible  Explanation.”  That  skill  also  involves  the 
application  of  a  generalization  process,  but  in  that  case  the  focus  is  on  telling  a  “story”  based  on  tying  events 
together.  A  major  subgoal  of  that  generalization  is  usually  to  make  some  sort  of  causal  inference. 

For  CT  Skill  4,  though,  the  thrust  is  to  generalize  from  a  specific  fact,  observation,  or  other  “piece  of  data”  to  a 
broader  class.  In  CT  Skill  3,  the  observed  instances  may  be  perfectly  clear  individually,  i.e.,  not  degraded  or  masked, 
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but  there  is  still  a  need  to  tie  them  together  where  the  story  is  used  to  derive  a  causal  inference.  But  in  CT  Skill  4,  we 

are  focused  on  isolated  instances  from  which  we  make  a  more  limited  (non-causal)  generalization. 

So,  the  type  of  generalizations  that  are  not  addressed  in  this  skill  include  generalizing  in  order  to: 

•  make  a  causal  inference 

•  infer  intent 

•  create  a  narrative  story 

•  derive  some  type  of  an  explanation 

Flow  of  Training 

The  training  flow  will  follow  four  elements,  which  include: 

1.  Describe  the  observation  or  instance.  This  involves  describing  the  observed  instance  in  terms  of  its  relevant 
features  and  details. 

2.  Identify  multiple  classes  that  the  observation  might  be  an  instance  of.  This  involves  coming  up  with 
alternative  categories  to  which  the  instance  might  be  generalized. 

3.  Evaluate  fit  to  multiple  classes.  This  involves  comparing  the  features  and  details  of  the  observed  instance  to 
each  of  the  possible  classes  and  determining  which  one  fits  the  best. 

4.  Identify  how  instance  could  change  over  time  to  fit  another  class.  Once  an  observation  is  classified,  a 
determination  is  made  concerning  how  that  classification  might  change  over  time. 

Training  Exercises 

A  variety  of  exercises  will  be  utilized  in  the  training  module. 

1 .  GUT  CHECK  is  a  technique  that  will  require  the  student  to  compare  his  or  her  intuitions  against  a  more 
systematic  feature  check.  The  technique  is  designed  to  show  that  while  one  can  certainly  use  their  instincts 
to  make  an  observation  and  classification  quickly,  such  judgments  will  result  in  missing  key  features  and 
focusing  on  salient  (i.e.,  noticeable)  dimensions  at  the  expense  of  more  subtle,  but  possibly  more  important 
dimensions. 

2.  DEVIL’S  IN  THE  DETAILS  is  a  technique  to  teach  students  how  to  tell  if  all  the  critical  features  in  an 
observed  instance  have  been  examined.  It  is  designed  to  help  make  one’s  observations  as  complete  as 
possible,  by  forcing  students  to  list  as  many  features  as  possible,  even  ones  that  do  not  seem  to  be 
important. 

3.  LOOK  BOTH  WAYS  is  a  technique  that  is  designed  to  overcome  priming  effects  and  other  attention- 
getting  properties  of  stimuli  that  tend  to  “capture”  one’s  focus  and  make  us  pick  the  first  classification  that 
comes  to  mind  rather  than  consider  other  categories  that  could  be  even  better  candidates. 

4.  WHAT  ELSE  COULD  IT  BE  is  a  technique  to  train  the  student  how  to  view  likely  candidate  classes.  It  is 
designed  to  create  that  “little  voice  in  your  head”  that  is  asking  what  other  class  the  observed  instance 
might  be  a  member  of 

5.  WHERE’S  WALDO?  is  a  technique  to  compare  an  observed  instance  with  many  other  classes.  It  is  based 
on  the  popular  children’s  book  series,  where  the  difficulty  of  finding  Waldo  depends  on  how  similar  to 
Waldo  are  all  the  other  non-Waldo  distracters.  The  more  similar  they  are,  i.e.,  the  one  more  features  they 
have  in  common  with  Waldo,  the  more  difficult  the  comparison  is  since  more  features  have  to  be  reviewed 
before  ruling  out  alternative  candidate  classes. 

6.  IF  IT  DOESN’T  FIT  YOU  MUST  ACQUIT  is  a  technique  for  teaching  students  to  know  how  and  when  to 
stop  checking  that  the  observed  instance  fits  as  a  member  of  one  or  more  target  classes.  The  emphasis  in 
this  training  will  be  to  instill  the  notion  that  sometimes  there  is  not  enough  evidence — i.e.,  not  enough 
features  are  associated  with  the  observed  instance — to  make  any  type  of  classification  with  confidence. 

7.  DO  YOU  FEEL  LUCKY?  is  a  technique  designed  to  teach  students  how  to  assess  the  confidence  level  they 
place  in  their  original  classification.  It  is  specifically  intended  to  instill  in  students  the  need  to  recheck  their 
classification  whenever  their  confidence  level  is  low. 

8.  THERE’S  NO  FREE  LUNCH  is  a  technique  that  shows  that  it  can  be  costly  not  to  recheck  one’s  original 
classification  at  some  later  point  in  time.  The  training  is  oriented  around  a  speeded  presentation  of  material, 
in  which  there  is  a  cost  (in  terms  of  limited  time  available)  associated  with  stopping  the  process  and 
rechecking  one’s  original  classification. 
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CT  SKILL  5:  MONITOR  AND  CONTROL  VISUALIZATION  AND  EVALUATE  ITS 
PRODUCTS 

WHAT  THIS  SKILL  IS  ABOUT 
Skill  Definition 

This  critical  thinking  skill  is  not  visualization1  per  se.  Rather,  it  is  the  ability  to  monitor  and  control  the  visualization 
process,  and  evaluate  the  mental  images  produced  by  the  visualization  process. 

Description  of  Skill 

Before  consideration  of  this  CT  skill,  it  is  important  to  understand  the  process  of  visualization,  its  purpose,  how  it 
benefits  the  individuals  who  use  it,  and  its  weaknesses.  Visualization  is  used  in  the  military  planning  process,  such 
as  in  the  Military  Decision  Making  Process  (MDMP)  and  during  operations.  It  involves  the  ability  to  use  mental 
images  to  “see”  past,  present,  and  future  military  situations,  and  also  to  predict  how  situations  may  evolve.  Thus, 
visualization  involves  both  the  creation  of  static  mental  images  as  well  as  the  ability  to  mentally  animate  these 
images  over  some  internally  consistent  timeline.  It  allows  military  personnel  to  anticipate  potential  conflicts  or 
problems,  allocate  resources  where  they  are  most  needed,  and  form  expectations  of  future  events.  Visualization  is  an 
individual  phenomenon,  but  through  communication,  its  products  may  be  shared  among  team  members  to  develop  a 
shared  visualization  of  a  situation. 

The  mental  images  created  by  the  visualization  process  are  produced  automatically  without  conscious  control  or 
deliberation.  They  are  created  from  past  experiences  and  knowledge  about  the  world.  Extensive  past  experience 
tends  to  develop  rich,  varied,  and  accurate  visualizations.  However,  extensive  past  experience  can  unduly  limit  and 
constrain  visualization,  producing  stereotypic  visualizations,  especially  for  novel  situations.  Past  experience  can  also 
set  up  expectations  and  assumptions  that  are  inaccurate  when  applied  to  novel  situations,  or  even  situations  that  are 
only  slightly  different  from  those  experienced  in  the  past.  In  other  words,  past  experience  is  highly  beneficial  to 
visualization;  but,  it  can  be  misleading.  This  training  will  provide  skills  to  maximize  the  value  of  experience  while 
minimizing  the  risk. 

This  CT  skill  serves  the  purpose  of  monitoring  and  controlling  the  visualization  process.  It  serves  as  a  meta- 
cognitive  check  on  the  visualization  process  but  It  is  not  about  creating  the  visualizations  per  se.  If,  for  example,  the 
visualization  process  focuses  on  irrelevant  details  of  the  situation,  the  ability  to  reflect  on  one’s  visualization  and 
then  consciously  redirect  it  is  one  aspect  of  this  CT  skill.  Or,  if  the  visualized  expectations  are  inaccurate,  this  CT 
skill  provides  a  check  that  may  serve  to  eliminate  or  modify  those  expectations.  In  this  way,  this  CT  skill  serves  the 
purpose  of  evaluating  the  mental  images  produced  by  the  visualization  process.  In  summary,  CTS  5  is  the  ability  to 
monitor,  control,  and  evaluate  the  process  and  products  of  visualization. 

Battle  Command  Tasks  Associated  with  Skill 

The  target  battle  command  tasks  for  which  this  skill  is  required  are  (1)  develop  scheme  of  maneuver  (2)  develop 
CO  As,  and  (3)  war  game  CO  As.  However,  this  skill  also  facilitates  many  other  military  tasks  including,  but  not 
limited  to: 

1 .  The  development  of  contingency  plans,  alternative  actions  that  might  be  taken  if  certain  events  transpire. 

2.  One’s  actions  during  the  mission  by  increasing  the  speed  of  response,  helping  to  select  appropriate 
responses,  and  helping  to  problem  solve  when  unforeseen  events  or  situations  occur. 

3.  Sensitization  to  relevant  cues  in  the  environment  related  to  danger  or  other  factors  that  may  affect  the 
mission. 

4.  Understanding  of  the  situation. 

5.  Communication  among  the  staff  or  team  members  in  the  planning  stage  as  well  as  the  execution  stage.  In 
the  planning  stage,  people  attempt  to  sync  their  visualizations  of  the  situation,  their  actions  as  a  team,  their 
vision  of  the  desired  end  state,  their  expectations  of  what  might  happen,  and  their  concerns  about  events 
that  may  interfere  with  successful  completion  of  the  mission.  In  the  execution  stage,  the  team  can  capitalize 

Definition  of  Visualization:  Visualization  is  the  process  of  creating  a  static  or  dynamic  mental  image  that 
has  pictorial  qualities  in  one’s  mind  of  a  past,  current  or  future  situation. 
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on  the  fact  that  they  share  a  visualization  because  it  allows  them  to  interpret  events  in  the  same  way  so  as  to 
better  respond  as  a  team,  i.e.  supporting  each  other’s  efforts,  anticipating  each  other’s  actions,  etc. 
Importantly,  this  can  be  done  even  when  they  are  geographically  separated.  It  is  a  process  that  supports 
shared  understanding  among  team  members. 

6.  Updating  of  a  strawman  model  of  the  situation  as  the  situation  unfolds.  This  updating  function  is  essential 
to  communication  to  other  team  units  and  to  command  so  as  to  “paint  the  picture”  for  the  commander 

Types  of  Errors 

A  variety  of  errors  commonly  occur  when  executing  the  critical  thinking  skill  of  monitoring  and  controlling 
visualization  and  evaluating  its  products: 

1.  Failure  to  visualize  key  variables  during  an  on-going  situation;  and  instead  visualize  those  that  are  salient 
or  more  accessible 

2.  Failure  to  check  that  the  implications  of  their  visualizations  are  consistent  with  the  actual  situation  at  hand. 

3.  Intrusion  of  past  experience  dominates  the  visualization,  which  establishes  over  confidence  and  results  in  a 
failure  to  identify  important  ways  in  which  the  new  situation  differs  from  past  situations. 

4.  Failure  to  consider  the  second  and  third  order  effect  of  actions. 


Learning  Objectives  for  this  Skill 

The  training  purposely  will  not  cover  visualization  per  se,  i.e.,  the  process  of  creating  visualization,  but  will  cover 
aspects  of  visualization  products  related  to  CT.  The  focus  for  the  training  on  this  skill  is  on  two  components  of 
visualization: 

1 .  The  student  shall  gain  skill  in  checking  expectations  and  assumptions  concerning  visualized  images  of  past, 
present,  and  future  experience,  and 

2.  The  student  shall  gain  skill  in  monitoring  his/her  visualizing  of  second  and  third  order  (indirect)  effects. 
Enabling  objectives  include: 

1 .  The  student  shall  gain  an  understanding  of  the  tendency  to  base  mental  images  on  previous  experience  and 
knowledge,  and  how  the  resulting  mental  images  may  be  inaccurate. 

2.  The  student  shall  increase  his/her  sensitivity  to,  and  recognition  of,  their  own  tendency  to  base  mental 
images  on  previous  experience  and  knowledge. 

3.  The  student  shall  gain  skill  in  checking  expectations  and  assumptions  that  may  underlie  his/her  mental 
images. 

4.  The  student  shall  gain  skill  in  checking  the  accuracy  of  his/her  mental  images. 

5.  The  student  shall  gain  an  understanding  of  the  tendency  to  avoid  visualizing  second  and  third  order  effects. 

6.  The  student  shall  gain  skill  in  visualizing  second  and  third  order  effects. 

Skill  Boundaries 

The  most  closely  related  skill  is  CTS  4,  Generalize  from  specific  instances  to  broader  classes.  CTS  4  develops  the 
ability  to  (in  real  time)  recognize  and  then  classify  specific  observations,  facts,  incidents,  or  events  into  a  broader 
class.  The  stimulus  material  for  CTS  4  is  often  pictorial.  CTS  5,  in  contrast,  considers  images  that  are  produced 
mentally,  where  the  originating  stimuli  might  not  necessarily  have  been  external  pictorial  stimuli. 


Flow  of  Training 

1.  The  training  will  begin  with  an  interactive  experience  that  shows  the  student  how  his/her  expectations  of  how 
something  looks  (images  of  past,  present,  or  future  experiences)  may  be  inaccurate.  Integral  to  the  training 
is  the  notion  that  the  inaccuracy  will  be  due  to  an  over  reliance  on  previous  experience  and  (1)  the  failure  to 
realize  and  acknowledge  that  an  expected  image  may  be  wrong,  and  (2)  the  failure  to  make  checks  and  seek 
additional  information  about  how  that  image  may  be  wrong.  Hence,  the  first  experience  contained  in  the 
first  few  pages  will  involve  looking  at  a  set  of  pictures  (training  set),  visualizing  scenes  related  to  this  first 
set,  viewing  a  set  of  “test”  pictures  and  selecting  those  that  they  believe  are  of  the  training  set.  This  first 
experience  will  be  made  intentionally  extremely  difficult  so  that  the  student  will  make  errors;  the  point  of 
which  is  to  increase  the  student’s  awareness  of  the  tendency  to  make  this  error. 
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A  micro-process  detail  describing  the  task  and  kinds  of  errors  we  expect  people  would  make  follows.  First, 
we  expect  people  to  look  at  a  set  of  pictures  at  time  t  and  then  create  an  image  of  what  happens  at  time 
t+Y?  Errors  might  occur  because  (1)  the  pictures  cannot  be  a  complete  representation  of  the  environment, 
(2)  they  may  lead  students  to  believe  one  thing  (e.g.,  movement  in  a  certain  direction),  when  in  fact  there’s 
evidence  in  the  picture  set  that  (e.g.,  a  barrier  at  the  top  of  the  hill)  that  movement  must  occur  in  a  different 
direction,  and  (3)  the  training  set  will  not  include  all  militarily  important  information  contained  in  the 
situation.  Note  that  you  one  could  invoke  a  visualization  from  non-pictorial  stimuli,  such  as  a  spot  report. 
For  example,  the  subject’s  task  could  be  to  visualize  what  the  enemy  will  be  doing  at  some  time  later  based 
on  a  spot  report. 

2.  Then,  there  will  be  a  series  of  exercises  that  provide  an  opportunity  for  the  student  to  make  similar 
predictions  about  how  something  will  look.  These  exercises  will  be  easier  and  will  allow  students,  if  they 
do  the  appropriate  checking  on  their  initial  images  (whatever  that  checking  may  amount  to)  will  do  better 
than  if  they  do  not.  We’ll  want  maybe  3  exercises  of  this  sort,  if  time  allows. 

3.  Then  we’ll  transition  to  the  second  and  third  order  effects  section.  The  ability  to  recognize  and  predict 
second  and  third  order  effects  depends  on  the  ability  covered  in  the  first  section  of  the  training.  If  a 
person’s  images  are  based  on  experience  and  they  go  unchecked,  they  will  be  unable  to  accurately  predict 
second  and  third  order  effects. 

4.  Then  we’ll  provide  exercises  on  predicting  second  and  third  order  effects,  focusing  on  the  images 
generated. 

5.  Training  will  end  with  summary  of  what  they  might  have  gotten  out  of  it  and  where  it  might  be  applicable. 

6.  Post  test. 

Training  Exercises 

The  training  will  include  the  following  exercises,  broadly  described. 

Training  Exercise  Set  1:  Check  on  Images — Downside  of  Experience 

Present  students  with  a  set  of  5  to  10  pictures  of  a  city,  family,  OR  type  of  structure.  The  pictures  should  be 
consistent  with  one  another  with  not  much  variability.  Students  will  then  be  instructed  to  visualize  other  scenes  from 
the  city,  which  may  be  cued  by  specific  verbally  delivered  instructions,  or  may  be  generally  instructed.  Then, 
students  are  presented  with  a  test  set  of  pictures,  5  to  7  of  them.  Some  of  the  pictures  would  be  from  the  same 
category  of  pictures  in  the  study  set,  some  from  a  different  category.  The  ones  from  different  category  would  be 
similar  but  would  have  something  that  indicated  they  weren't  from  the  same  category.  The  student’s  job  will  be  to 
pick  which  of  the  pictures  best  matches  their  visualization. 

Training  Exercise  Set  2:  Indirect  Effects 

Present  student  with  a  historical  example,  one  that  is  not  all  that  well  known,  in  which  a  decision  was  made  and  as  a 
result  of  that  decision,  there  were  a  set  of  direct  effects  (1st  order).  Then,  those  direct  effects  in  turn  caused  a  set  of 
additional  outcomes  that  are  known  (2nd  order).  Those  second  order  effects  in  turn  caused  a  set  of  third  order  effects 
that  history  documents. 

1 .  Students  are  first  asked  to  visualize  as  many  2nd  order  effects  as  they  can.  THEN,  students  are  given  a  set  of 
outcomes  that  MAY  have  been  the  direct  result  of  the  direct  effects  (2nd  order).  Student’s  task  is  to  select 
the  ones  that  best  match  their  visualized  second  order  effects.  We  could  then  give  them  feedback  about 
their  choices  (tell  them  which  actually  were  2nd  order  effects  and  which  were  not).  There  will  be  no 
question  about  the  accuracy  of  the  “correct”  answers  because  history  would  document  the  facts.  The 
exercise  will  generate  awareness  of  how  many  different  kinds  of  things  can  happen  as  a  result  of  the  effects 
of  actions,  and  the  effects  of  effects. 

2.  An  additional,  related,  exercise  will  be  used  after  the  first.  It  will  present  the  set  of  second  order  effects 
before  describing  the  direct  first  order  effects,  but  after  the  brief  description  of  the  situation  and  action. 
Student’s  job  would  then  be  to  identify  which  second  order  effects  were  the  direct  result  of  a  set  of  first 
order  effects,  which  would  be  displayed.  This  is  the  Jeopardy  version  of  the  first  exercise  because  you  get 
the  answer  before  the  question,  or  in  this  case,  the  effect  before  the  cause. 
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CT  SKILL  6:  CHALLENGE  ONE’S  BIAS 


WHAT  THIS  SKILL  IS  ABOUT 
Skill  Definition 

This  critical  thinking  skill  is  the  ability  to  consistently  reevaluate  one’s  current  view  of  the  situation  for  prejudice  or 
bias  as  new  information  is  received.  Specifically,  it  is  the  ability  to  overcome  a  built-in  cognitive  bias  that  all  people 
have — the  tendency  to  collect  and  attend  to  information  that  confirms  a  hypothesis  or  belief  that  we  have,  at  the 
expense  of  potentially  disconfirming,  negative  evidence. 

Description  of  Skill 

Before  considering  this  CT  skill  in  detail,  it  is  important  to  understand  the  erroneous  tendency  it  is  designed  to 
overcome,  confirmation  bias.  This  bias  comes  about  as  a  natural  result  of  forming  hypotheses  or  beliefs  concerning 
upcoming  events.  Our  hypotheses  help  guide  us  in  the  search  for  new  information  so  that  our  perceptions  are  not 
random.  However,  the  price  we  pay  is  that  our  senses  are  tuned  or  primed  to  look  for  those  aspects  in  the 
surrounding  environment  that  would  confirm  the  validity  of  our  hypothesis.  This  tendency,  which  everyone  has,  can 
result  in  “cognitive  tunnel  vision,”  wherein  we  either  fail  to  detect  any  conflicting  information,  or  if  we  do  notice  it, 
we  “explain  it  away”  as  not  being  relevant  to,  or  cause  for,  changing  our  current  beliefs. 

A  classic  example  of  confirmation  bias  is  found  in  the  1988  incident  onboard  the  USS  Vincennes,  which  had  tragic 
consequences.  During  a  prologue  to  the  1990  Iraqi  Gulf  War,  our  ships  were  routinely  patrolling  in  the  Persian  Gulf 
waters.  The  captain  of  the  Vincennes  had  formed  a  hypothesis  that  an  approaching  aircraft  was  hostile.  Even  though 
some  on  his  crew  were  uneasy  with  that  rapid  assessment,  the  captain  failed  to  consider  alternative  hypotheses  (e.g., 
that  the  aircraft  was  an  Iraqi  commercial  airliner).  As  a  result,  personnel  in  the  ship’s  Combat  Information  Center  (1) 
failed  to  recognize  that  the  aircraft’s  altitude  was  increasing,  indicating  non-hostile  intentions;  instead  it  was 
assumed  that  its  approach  profile  was  descending;  (2)  incorrectly  perceived  the  airliner’s  IFF  transponder  was  set  to 
Mode  II  (military)  when  it  in  fact  was  set  on  Mode  III  (commercial);  and  (3)  failed  to  realize  that  commercial  traffic 
patterns  in  their  area  were  such  that  a  prior  takeoff  in  that  time  period  would  put  an  aircraft  in  a  position  consistent 
with  the  approaching  aircraft.  Thinking  the  radar  screen  contact  was  a  fighter-sized  aircraft,  they  launched  a  missile 
attack  killing  everyone  on  Flight  655.  The  captain’s  tunnel  vision,  reflecting  a  confirmation  bias,  caused  him  to  both 
(1)  fail  to  look  for  any  evidence  that  might  disconfirm  his  belief  (that  the  approaching  aircraft  was  hostile)  and  (2) 
interpret  all  new  information  as  consistent  with  his  hypothesis.  The  results  were  catastrophic. 

We  can  identify  the  logical,  mental  steps  that  one  must  go  through  to  properly  form  and  test  one’s  hypothesis  about 
some  aspect  of  the  world.  These  five  steps  are  shown  below. 


Step  0:  Develop 
a  working  hypothesis 
about  the  situation 


Step  1 :  Identify 
information  elements 
that  would  disprove 
the  hypothesis 


Step  2:  Seek  out 
that  disconfirming 
evidence 


Step  3:  Interpret 
the  evidence  as 
disconfirming 
once  found 


Step  4:  Revise 
hypothesis  on  basis 
of  new  evidence 


We  start  by  developing  a  working  hypothesis  about  something  in  the  world  that  has  not  happened  yet,  but  might 
(Step  0).  This  hypothesis  will  then  guide  us  in  the  types  of  information  we  look  for  so  that  our  environment  does  not 
seem  random.  We  cover  this  step  in  other  skills  so  we  will  not  address  it  here.  Instead,  CTS  6  will  assume  that  this 
step  has  already  been  formed.  In  Step  1,  it  is  important  to  identify,  in  advance,  the  types  of  information  that,  if 
found,  would  provide  disconfirming  evidence  against  our  hypothesis.  The  Captain  of  the  Vincennes  failed  to 
perform  this  step.  Had  he  done  so,  he  would  have  focused  on  cues  like  changes  in  altitude,  IFF  mode  settings,  and 
the  like.  In  Step  2,  the  critical  thinker  must  put  in  place  actions  that  will  help  ensure  that  potentially  conflicting 
information  is  collected  if  it  arrives.  In  this  current  example,  this  would  mean  having  people  posted  in  the  CIC 
whose  duty  includes  continually  watching  for  signs  that  the  prevailing  hypothesis  might  be  wrong.  It  would  mean 
monitoring  items  such  as  altitude,  IFF  mode  settings,  commercial  air  traffic  so  that  if  counter-evidence  is  available, 
it  isn’t  missed. 


Once  counter-evidence  is  found,  it  should  be  reviewed  fairly  and  not  “explained  away”  (Step  3).  In  the  present 
example,  this  would  mean  that  if  the  CIC  had  been  monitoring  the  aircraft’s  altitude  and  spotted  the  increase,  it 
would  be  interpreted  clearly  (i.e.,  as  something  that  means  no  harm),  as  opposed  to  some  conjecture  like:  “he’s 
increasing  altitude  to  ensure  that  he’s  outside  his  ‘frag’  envelope  once  he  releases  his  bombs.”  The  final  step  (Step 
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4)  involves  actually  revising  one’s  hypothesis  once  disconfirming  evidence  has  been  specified,  collected,  and 
interpreted.  This  will  typically  involve  weighing  evidence  from  multiple  sources,  not  just  one,  and  making  the  most 
objective  assessment  possible  given  time  and  resource  constraints. 

The  skill  we  are  describing  here  is  not  easy,  since  everyone  is  tuned  to  look  for  evidence  that  confirms  rather  than 
disconfirms  our  beliefs.  This  bias  is  a  known  problem  in  a  number  of  professions,  not  just  the  military,  where 
evidence  must  be  collected  and  interpreted.  Thus,  we  see  confirmation  bias  crop  up  in  areas  such  as  medical 
diagnosis,  weather  forecasting,  and  legal  analysis.  To  compound  the  problem  further,  our  “cognitive  tunnel  vision” 
tends  to  get  even  narrower — we  miss  even  more  disconfirming  evidence — when  we  are  stressed,  overloaded,  and 
under  time  pressure.  The  techniques  we  provide  under  this  skill  are  designed  to  “widen”  one’s  cognitive  vision  so 
that  important  disconfirming  information  is  identified,  collected,  and  interpreted  properly. 

Battle  Command  Tasks  Associated  with  Skill 

The  target  battle  command  task  for  which  this  skill  is  required  is  changing  own-unit  plans  based  on  new  tactical 
input.  However,  this  skill  also  facilitates: 

1.  Development  of  the  commander’s  critical  information  request  (CCIR),  in  which  information  “tripwires”  are 
identified  to  help  the  commander  know,  as  soon  as  possible,  if  conditions  on  the  battlefield  (e.g.,  enemy 
location,  visibility,  dispersal  of  friendly  tactical  assets)  have  changed. 

2.  Tracking  the  battlefield,  where  the  ability  to  objectively  identify  new  information,  even  if  it  is  at  odds  with 
what  was  expected,  is  needed  to  determine  if  a  tactical  force  needs  to  be  reconstituted  or  the  mission 
objectives  need  to  be  modified. 

3.  Assessing  the  situation,  where  planners  and  analysts  need  to  know  which  new  information  will  confirm  or 
disconfirm  prior  assessments,  and  have  the  means  to  collect  such  information. 

4.  Inferring  status  of  enemy  forces,  where  a  clear,  unbiased  assessment  of  enemy  intent  is  needed  in  order  to 
determine  the  most  effective  countermeasures. 

5.  Developing  a  common  operating  picture  (COP).  The  commander  and  staff  must  view  all  new  information, 
even  unexpected  and  “conflicting”  (with  some  currently  held  belief)  information,  with  a  “clear  eye”  so  they 
may  arrive  at  an  assessment  that  can  be  synched  with  commanders  of  other  units  who  may  not  be  holding 
the  same  pre -operational  opinions.  A  COP  can  only  arise  if  all  commands  view  new,  incoming  information 
with  an  unbiased  mindset. 

Types  of  Errors  that  commonly  occur  when  executing  this  skill: 

1.  Failure  to  identify  the  evidence  that  would  be  needed  to  disconfirm  a  currently  held  view  or  belief. 

2.  Failure  to  identify,  when  available,  evidence  that  runs  counter  to  one’s  current  beliefs  or  hypothesis  about  a 
given  situation. 

3.  Failure  to  actively  seek  out  information  that  would  disconfirm  a  currently  held  view  or  belief. 

4.  Failure  to  identify  how  potentially  conflicting  information  could  be  obtained. 

5.  Failure  to  interpret  conflicting  information  as  in  fact  conflicting,  but  instead  interpreting  it  as  consistent 
with  the  originally  held  belief  or  hypothesis. 

6.  Failure  to  integrate  conflicting  information  into  the  currently  held  hypothesis  and  revise  one’s  beliefs 
accordingly. 

Learning  Objectives  for  this  Skill 

This  training  module  will  focus  on  two  terminal  learning  objectives  that  constitute  aspects  or  components  of  this  CT 
skill.  Upon  completing  this  training  module  the  student  shall  gain  skill  in: 

1 .  Identifying  information  that  will  either  confirm  or  disconfirm  a  presently  held  belief  or  hypothesis,  and 

2.  Collecting  and  interpreting  information  that  can  be  used  to  disconfirm  a  presently  held  belief  or  hypothesis. 
Enabling  objectives  include: 

1 .  The  student  shall  gain  an  understanding  of  the  tendency  or  bias  to  ignore  information  that  is  inconsistent 
with  a  currently  held  belief  or  hypothesis. 

2.  The  student  shall  gain  an  understanding  of  the  tendency  or  bias  to  seek  out  information  that  is  consistent 
with  a  currently  held  belief  or  hypothesis. 
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3.  The  student  shall  gain  an  understanding  of  the  tendency  or  bias  to  interpret  neutral  or  irrelevant  information 
as  being  consistent  with  a  currently  held  belief  or  hypothesis. 

4.  The  student  shall  gain  skill  in  correctly  interpreting  new  information  as  either  consistent  with  or 
inconsistent  with  a  currently  held  belief  or  hypothesis. 

5.  The  student  shall  gain  skill  in  revising  a  currently  held  belief  or  hypothesis  so  that  it  is  consistent  with  the 
information  that  is  available. 

Skill  Boundaries 

There  are  two  closely  related  skills  to  this  skill.  CTS  4,  Generalize  from  Specific  Instances  to  a  Broader  Class, 
develops  the  skill  to  recognize  (in  real-time)  information  items  as  they  arrive  and  then  classify  them  into  useful 
categories.  Some  of  the  categories  discussed  in  that  skill  will  serve  as  working  hypotheses  (or  beliefs)  for  this  skill. 
CTS  3,  Develop  Explanation  that  ties  the  Evidence  together,  is  a  skill  that  underlies  Step  0 — Develop  a  Working 
Hypothesis — in  this  skill.  Consequently,  the  first  step  in  the  present  skill  will  receive  little  coverage  as  it  is  already 
addressed  in  CTS  5. 

Flow  of  Training 

1.  The  training  will  begin  with  an  example  of  how  confirmation  bias  can  operate  insidiously.  We’ll  use  the 
example  where  the  user  sees  the  numbers  2,  4,  6  and  we  ask  them  to  guess  what’s  next.  The  underlying  rule 
is  “the  numbers  increase.”  However,  because  the  pattern  looks  so  mathematically  precise,  most  people  tend 
to  focus  on  rules  like  “it  goes  up  by  two.”  They  then  interpret  each  next  number,  which  might  be  a  7,  or  9, 
as  suggestive  of  some  other,  increasingly  more  complex  rule.  This  exercise  will  engage  the  student  in  an 
experience  of  how  confirmation  bias  can  put  “cognitive  blinders”  on  someone  to  the  point  that  they  fail  to 
objectively  review  new  evidence. 

2.  As  an  optional  link,  we  will  provide  a  multi-media  illustration  of  how  confirmation  bias  has  caused 
disasters  in  military  settings.  We’ll  use  as  our  example  a  fratricide  case  history  that  occurred  subsequent  to 
the  first  Gulf  War.  This  involved  a  breakdown  in  coordination  between  Air  Force  and  Army  airborne  units 
while  patrolling  the  skies  over  Iraq  (Operation  Provide  Comfort)  in  which  a  two-ship  of  F- 15s  ignored 
available  counter-evidence  and  shot  down  an  Army  Blackhawk  helicopter.  Specific  details  from  that  case 
history  will  be  amplified  and  shown  to  link  directly  to  problems  with  confirmation  bias. 

3.  We’ll  then  go  into  the  first  set  of  exercises,  which  will  collectively  address  the  problems  associated  with 
Step  1,  Identifying  Potentially  Conflicting  Information.  An  introductory  exercise  will  expose  students  to  the 
different  forms  that  a  decision-making  bias  might  take,  to  increase  their  self  awareness.  Then  the  main 
exercises  will  require  students  to  reflect  on,  then  list,  the  type  of  evidence  they  would  need  to  change  their 
hypothesis  about  a  situation.  The  hypothesis  they  will  be  forming  will  vary,  from  a  simple  assumption 
about  enemy  intent  or  avenue  of  approach,  to  more  elaborate  battle  plans.  The  exercises  would  be  presented 
in  the  form  of  a  what-if  game.  Part  of  the  skill  training  will  involve  listing  a  series  of  information  elements 
and  asking  the  student  to  state  whether  they  confirm  or  disconfirm  that  hypothesis.  This  could  be  done  by 
dragging  individual  pieces  of  evidence  into  the  confirm  box  or  disconfirm  box.  As  a  supplemental  exercise, 
we’ll  have  students  attempt  to  “widen  their  cognitive  tunnel”  by  selecting  information  they  would  want  (to 
confirm  or  disconfirm)  from  a  lengthy  list  of  possibilities.  Once  again,  the  “hypothesis”  being  tested  might 
start  off  being  a  simple  piece  of  evidence  (e.g.,  a  CCIR)  and  then  become  more  elaborate,  such  as  a  set  of 
(logically,  empirically)  inter-related  assumptions  or  even  a  full-blown  plan. 

4.  Then  we’ll  transition  into  the  second  set  of  exercises,  which  will  collectively  address  the  problems 
associated  with  Step  2,  Collecting  Disconfirming  Evidence.  These  exercises  will  have  students  actively 
select  each  piece  of  new  evidence  as  it  comes  in  rather  than  passively  observe  all  evidence.  Emphasis  will 
be  placed  on  adopting  an  active,  controlling  orientation  toward  all  features  in  the  evidence,  including 
negative  (i.e.,  the  absence  of  confirming  evidence).  In  some  cases,  students  will  be  instructed  to  take  an 
“outside  view”  when  they  collect  the  evidence,  by  addressing  the  role  and  influence  of  contextual 
information  directly.  They  will  learn  to  view  context  as  part  of  the  stimulus  environment  they  are 
addressing. 

5.  We  will  then  administer  the  third  set  of  exercises,  which  collectively  address  problems  associated  with 
interpreting  potentially  disconfirming  evidence  (Step  3)  and  revising  one’s  hypothesis  based  on  the  new 
evidence  (Step  4).  The  exercises  will  all  have  students  operate  a  “doubt  meter,”  in  which  they  state  their 
current  confidence  in  their  hypothesis  and  update  that  confidence  statement  as  each  new  piece  of  evidence 
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6. 


arrives.  On  occasional  trials,  we’ll  have  the  student  “sift  through  trash,”  by  reviewing  evidence  that  they 
had  previously  “explained  away.”  They’ll  use  this  review  to  recalibrate  their  doubt  meter. 

Training  will  end  with  a  summary  of  what  they  might  have  gotten  out  of  it  and  where  it  might  be 
applicable. 

7.  Post  test. 

Training  Exercises 

The  training  will  be  organized  around  3  classes  of  techniques,  each  class  designed  to  counteract  our  tendency  to 
focus  on  confirming  evidence  to  the  exclusion  of  potentially  disconfirming  evidence.  These  techniques  will  be 
assembled  into  a  set  of  exercises  that  address  one  of  the  four  steps  (excluding  Step  0)  in  the  sequence  depicted 
above. 

Training  Exercise  Set  1:  Identifying  Potentially  Disconfirming  Information 

Introductory  Exercise:  Self  Awareness.  Provide  several  compelling  examples  of  decision  making  biases,  using 
simple  real-world  example  stimuli,  to  increase  students’  self-awareness  of  the  presence  of  confirmation  bias  in  all 
that  we  do.  Other  related  biases  (overconfidence,  positive  testing)  will  be  illustrated  as  well. 

Main  Exercise  Thread:  What-if  Game.  Students  will  start  with  a  particular  hypothesis  as  a  given.  The  hypothesis 
will  at  first  be  a  single-item  hypothesis,  and  then  progress  into  complex,  multi-item  hypotheses.  For  each  trial, 
students  will  be  asked  to  play  a  what-if  game  by  considering  different  pieces  of  evidence  so  that  IF  they  appeared, 
would  they  confirm  or  disconfirm  their  hypothesis.  They  will  drag  these  what-if  items  into  either  a  confirm  or  a 
disconfirm  box.  Scores  will  be  posted  and  times  provided  as  feedback. 

Exercise  Supplement:  Widen  your  Cognitive  Tunnel.  On  some  of  the  exercises,  students  will  attempt  to  broaden  their 
perspective  (widen  their  cognitive  tunnel)  by  considering  a  larger  than  normal  range  of  information  items.  They  will 
do  this  by  reviewing  a  lengthy  list  of  potential  pieces  of  evidence  for  a  given  hypothesis.  This  widening  will  be 
designed  to  help  them  form,  and  then  consider,  alternative  hypotheses  to  their  original  hypothesis. 

Training  Exercise  Set  2:  Actively  Collect  Potentially  Disconfirming  Information 

Main  Exercise  Thread:  Seeking  Evidence.  Students  will  be  given  some  hypothesis  and  then  asked  to  actively  seek 
out  individual  pieces  of  information  to  confirm  their  hypothesis.  The  focus  will  be  on  active  selection  of  information 
as  a  conscious  choice,  where  they  will  be  required  to  reflect  on  why  they  are  choosing  to  seek  out  certain  pieces  of 
information.  Cues  and  hints  will  be  embedded  in  the  search  materials  to  instill  in  students  the  importance  of  seeking 
out  disconfirming  evidence  as  well  as  confirming  evidence. 

Exercise  Supplement:  Outside  Viewing.  During  some  of  the  search  trials  students  will  be  shown  examples  where 
they  are  either  looking  at  the  information  from  the  “inside,”  in  which  contextual  information  is  not  considered,  or 
looking  at  it  from  an  “outside  view,”  where  contextual  information  is  considered.  These  exercise  supplements  will 
help  students  see  that  contextual  information  surrounding  an  event  is  part  of  the  stimulus  array  and  should  often  be 
considered  just  like  any  other  information. 

Training  Exercise  Set  3:  Interpreting  Disconfirming  Information 

Main  Exercise  Thread:  Doubt  Meter.  Students  will  be  given  a  hypothesis  and  asked  to  collect  information  to  test  it. 
As  they  collect  the  information,  they  will  be  required  to  state  their  current  confidence  level  in  their  hypothesis, 
where  this  confidence  level  will  be  updated  after  each  new  piece  of  information  comes  in.  The  metaphor  to  be  used 
will  be  a  “doubt  meter,”  expressed  as  a  yardstick  whose  length  the  student  can  manipulate  to  indicate  their  current 
level  of  confidence.  The  complexity  of  the  hypothesis  will  increase  progressively  over  the  course  of  the  exercise, 
starting  with  simple  single-item  hypotheses  (CCIR  present/absent)  to  multi-item  hypotheses  (enemy  approaches  via 
Avenue  A  then  changes  their  intent  from  neutral  to  hostile)  to  a  full-blown  battle  plan. 

Exercise  Supplement:  Sifting  through  Trash.  On  occasional  exercises,  as  students  review  individually-presented 
pieces  of  information,  they  will  be  asked  to  open  the  trash  to  look  at  previous  interpretations  that  they  had  discarded 
(or  “explained  away”).  The  training  is  designed  to  sensitize  students  to  the  dangers  of  discounting  potentially 
conflicting  evidence  prematurely.  They  will  recalibrate  their  “doubt  meters”  accordingly  after  each  review. 


B-17 


CT  SKILL  7:  EXAMINE  OTHER  PEOPLE’S  PERSPECTIVES 


WHAT  THIS  SKILL  IS  ABOUT 
Skill  Definition 

The  ability  to  examine  other  people’s  perspectives  is  the  ability  to  view  and  interpret  a  set  of  circumstances  from  the 
perspectives  of  different  individuals,  different  cultures/religions,  and  different  timeframes  (historical  perspective). 

Description  of  Skill 

The  ability  to  understand  other  people’s  perspectives  takes  conscious  and  deliberate  cognitive  effort.  Our  natural 
tendency  is  to  interpret  and  predict  others’  behavior  based  on  our  own  goals,  intentions,  values,  assumptions,  beliefs, 
roles  and  preferences.  For  this  reason,  it  is  very  common  to  misunderstand  another’s  perspective,  especially  when 
the  other  person  comes  from  a  different  culture  or  background  than  one’s  own.  It  is  almost  an  automatic  response  to 
ascribe  meaning  to  others’  actions  based  on  what  we  ourselves  would  mean  by  those  actions.  There  is  also  a  strong 
automatic  tendency  to  attribute  another  person’s  behavior  to  personal  qualities,  rather  than  the  circumstances  that 
surrounded  the  person. 

This  CT  skill  involves  placing  checks  on  automatic  attributions  about  another  person  to  carefully  and  consciously 
examine  his  or  her  perspective.  Individuals  who  possess  high  levels  of  this  skill  are  able  to  impede  the  automatic 
response,  and  focus  deliberate  thought  on  the  examination  of  other’s  perspectives.  As  a  result,  they  demonstrate  a 
better  understanding  of  the  other  person’s  goals  and  intentions  and  can  better  predict  their  future  behavior. 
Individuals  who  perform  poorly  in  this  skill  assume  that  their  immediate  interpretations  of  other’s  perspectives  are 
correct.  As  a  result,  their  attributions  are  frequently  incorrect. 

In  both  cooperative  and  adversarial  relationships,  understanding  others’  desires  and  goals  is  of  critical  importance. 

In  a  cooperative  relationship,  understanding  the  perspectives  of  colleagues,  superiors,  and  those  who  work  at  your 
direction  are  critical  to  harmonizing  effort  for  maximum  effect.  Understanding  an  adversary’s  perspective  is  critical 
to  predicting  future  behavior.  In  cooperative  and  adversarial  relationships,  there  are  several  major  sources  from 
which  the  future  behavior  can  be  predicted.  For  example,  past  behavior,  the  physical  capacities  and  constraints  the 
other  faces,  the  situation,  and  the  other’s  goals  are  sources  of  information  that  can  be  used  to  predict  future  behavior. 

One  may  understand  another  person’s  perspective  at  the  strategic  and  tactical  level.  At  the  strategic  level,  the  task 
involves  understanding  another  person’s  (team  member  or  enemy)  goals,  the  history  from  which  they  emerge,  and 
the  preferred  means  of  pursuing  them.  At  the  tactical  level,  this  skill  is  central  to  rapidly  adapting  to  evolving  events 
and  correctly  anticipating  what  the  other  is  most  likely  to  do  next. 

Battle  Command  Task  Associated  with  Skill 

The  most  salient  battle  command  tasks  that  demand  competence  in  this  skill  are  interpreting  reports  of  recent  enemy 
activities  in  an  area  of  interest  and  determining  enemy  intent.  Other  battle  command  tasks  that  involve  this  skill 
include:  infer  status  of  enemy  forces,  interpret  reports  of  enemy  disposition,  read  the  battlefield,  and  track  the 
battlefield,  and  assess  the  situation. 

This  skill  is  also  important  to  unit  effectiveness.  Even  though  any  given  Army  unit  shares  the  general  goal  or 
mission,  the  distribution  of  responsibilities  nearly  always  means  that  individual  team  members  will  have  different 
perspectives,  sub  goals,  information  needs,  etc.  The  ability  to  accurately  understand  other  team  members’ 
perspectives  greatly  enhances  team  cohesion  and  effectiveness. 

Types  of  Errors 

The  training  for  this  skill  will  target  four  general  and  common  types  of  error  in  applying  this  skill: 

1.  Failure  to  attribute  another’s  actions  or  motivation  to  situational  factors. 

2.  Failure  to  understand  and  integrate  the  assumptions  and  expectations  that  underlie  or  follow  from  the  goals 

3.  Misunderstanding  the  costs  the  other  is  willing  to  bear,  or  to  impose,  to  achieve  a  goal 

4.  Inaccurate  or  inadequate  assessment  of  the  uncertainty  of  each  of  the  three  issues  above 
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Learning  Objectives  for  this  Skill 

The  training  for  this  skill  targets  two  learning  objectives: 

1.  Dispassionately  understand  others’  goals,  motivations,  assumptions,  expectations,  and  the  costs  they  are 
willing  to  bear 

2.  Calibrate  the  uncertainty  in  your  assessment 
Enabling  objectives  include: 

1.  Recognizing  that  understanding  others’  perspectives  is  important  and  difficult. 

2.  Recognizing  that  the  adversary  may  have  meaningfully  different  values,  assumptions,  and  expectations 
than  oneself. 

3.  Recognizing  that  superficial  assessments  and  failure  to  learn  increases  risk  and  decreases  chances  of 
success. 

Skill  Boundaries 

The  most  closely  related  skills  are  CTS  3,  ‘Develop  an  explanation  that  ties  evidence  together  in  a  plausible  way’, 
and  CTS  6,  ‘Challenge  one’s  bias’.  Developing  an  explanation  might  require  seeing  things  from  another  person’s 
perspective.  Therefore,  there  is  some  overlap  between  CTS  3  and  CTS  7.  The  situation  for  which  an  explanation  is 
developed,  however,  might  not  involve  any  other  person.  Moreover,  developing  an  explanation  involves  other 
elements  such  as  tying  various  elements  of  a  situation  together,  integrating  the  circumstantial,  material,  and  personal 
elements,  and  making  testable  predictions.  Hence,  the  two  skills  can  be  distinguished. 

Challenging  one’s  bias  overlaps  with  examining  another’s  perspective  because  the  bias  one  needs  to  challenge  may 
be  about  another  person.  CTS  6,  however,  focuses  on  the  ability  to  reduce  confirmation  bias,  which  is  the  tendency 
to  seek  confirming  and  avoid  disconfirming  evidence.  Hence,  CTS  6  may  be  used  in  conjunction  with  CTS  7  when 
attempting  to  avoid  confirmation  bias  about  another’s  perspective.  However,  the  two  skills  are  based  in  avoiding 
different  types  of  errors  and  require  different  processes. 

Flow  of  Training 

1 .  The  student  is  presented  with  an  opening  example  that  demonstrates  that  there  are  alternative  ways  to 
interpret  others’  behaviors  and  perspectives,  and  that  some  ways  are  better  than  others.  They  are  given  a 
series  of  exercises  in  which  they  evaluate  various  reasons  why  an  individual  would  take  a  certain  action. 

2.  They  are  then  presented  with  a  series  of  exercises  in  which  they  are  required  to  evaluate  a  particular 
scenario  from  a  variety  of  different  individual’s  perspectives. 

3.  This  is  followed  by  the  delivery  of  information  about  the  fundamental  attribution  error,  in  which  people 
commonly  make  the  mistake  of  attributing  dispositional  causes  to  others’  behavior  but  situational  causes  to 
their  own  behavior.  This  is  also  called  the  person-situation  bias  in  this  training. 

4.  They  are  then  given  several  exercises  that  require  students  to  identify  indicators  that  they  might  not  be 
understanding  others’  perspectives. 

5.  A  series  of  exercises  follows  instruction  in  the  PMESII  method  used  by  the  military,  which  can  be  used  to 
better  understand  others’  perspectives. 

Training  Exercises 

The  training  will  include  the  following  exercises,  broadly  described. 

Training  exercise  Set  1:  What  does  it  look  like  from  someone  else’s  viewpoint? 

Training  Exercise  Set  2:  Identify  “we’re  not  on  the  same  page”  indicators. 
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CT  SKILL  8:  DECIDE  WHEN  TO  SEEK  INFORMATION  BASED  ON  ITS  VALUE 
AND  COST 

WHAT  THIS  SKILL  IS  ABOUT 
Skill  Definition 

This  skill  involves  the  ability  to  evaluate  the  need  for  new  information  in  terms  of  its  value  relative  to  its  potential 
costs  in  time,  resources  and/or  risk.  It  encompasses  the  ability  to  overcome  the  tendency  to  spend  too  much  time 
gathering  unnecessary  information,  as  well  as  the  tendency  to  act  too  quickly  without  gathering  and  analyzing 
sufficient  information. 

Description  of  Skill 

Is  sufficient  information  at  hand  to  make  an  informed  judgment,  to  make  a  decision  with  sufficient  certainty,  or  to 
develop  an  adequate  plan?  How  does  one  know?  What  human  limitations  must  be  overcome  in  acquiring  and 
employing  additional  information  that  is  appropriate  to  the  task?  What  is  the  relationship  between  the  acquisition  of 
additional  information,  the  accuracy  of  ensuing  plans  and  decisions,  and  the  levels  of  confidence  one  has  in  these 
plans  and  decisions?  What  types  of  additional  information  lead  to  increased  accuracy  in  judgments  and  what  types 
do  not?  These  questions  suggest  the  challenges  involved  in  developing  skills  needed  for  deciding  when  to  seek 
information  based  on  its  anticipated  value  versus  its  cost  of  acquisition. 

A  substantial  body  of  research  has  revealed  the  principal  challenges  entailed  in  seeking  the  appropriate  information. 
Moreover,  research  conducted  with  experts  in  fields  as  diverse  as  military  operations,  weather  forecasting,  medical 
diagnosis,  stock  market  analysis  and  horserace  handicapping  have  shown  that  the  same  challenges  are  encountered 
consistently.  Consequently,  a  critical  component  of  this  skill  is  the  capability  to  understand  and  to  meet  successfully 
these  challenges  that  will  certainly  be  encountered. 

The  first  principal  challenge  is  that  of  knowing  when  one  has  the  minimum  information  needed  to  make  an  informed 
judgment,  because  this  is  the  point  at  which  further  information  seeking  should  stop.  Additional  information 
obtained  after  this  point  typically  does  not  improve  the  correctness  of  decisions  or  the  accuracy  of  estimates.  Worse, 
the  additional  information  typically  results  in  unwarranted  increased  confidence  in  the  resulting  decision  or  estimate. 

The  second  principal  challenge  is  that  of  understanding  just  what  information  is  actually  being  employed  in  the 
planning-decision  process.  Even  experienced  planners  can  be  unaware  of  what  information  they  actually  use  during 
this  process.  Research  results  have  shown  that  most  do  not  realize  that  their  judgments  are  determined  by  a  relatively 
few  dominant  factors  rather  than  by  the  systematic  integration  of  all  available  information — that  is,  most  use  much 
less  of  the  available  information  than  they  think  they  do. 

A  simple  investigation  illustrates  these  two  challenges;  it  was  conducted  a  number  of  years  ago  by  decision 
researcher  Paul  Slovic  with  experienced  horserace  handicappers.  He  first  asked  each  handicapper  to  select,  from  a 
list  of  variables  employed  in  handicapping,  the  most  important  5,  10,  20  and  40  items  of  information  used  to 
handicap  a  horse  race — that  is,  which  he  would  want  to  use  if  limited  to  only  5  items  of  information  per  horse,  and 
similarly  if  limited  to  10,  20,  and  40  items.  He  then  gave  each  handicapper  true  data  on  horses  from  40  past  races 
and  asked  him  to  predict  the  results  of  the  race  four  times,  using  the  5,  10,  20,  and  40  items  of  information  for  each 
horse  for  each  race — thus  predicting  each  race  four  times.  Each  handicapper  also  indicated  his  degree  of  confidence 
in  his  predictions.  The  handicapper’s  predictions  were  as  accurate  with  5  items  of  information  as  they  were  with  10, 
20  or  40;  accuracy  did  not  increase  as  information  was  added.  Even  so,  the  handicappers  expressed  steadily 
increasing  confidence  in  their  predictions  as  the  amount  of  information  available  increased. 

But  what  about  the  type  of  information  that  is  added  beyond  the  minimum  required?  Would  not  some  types  of 
additional  information  contribute  potentially  more  than  others  to  the  accuracy  of  judgments?  There  are  two  types  of 
additional  information  that  are  not  likely  to  contribute  to  the  efficacy  of  the  process — additional  detail  about 
information  items  already  available  and  additional  information  items  beyond  the  minimum  required.  The  main  effect 
of  providing  supplementary  detail  about  and  consistent  with  information  already  available  and  addressed  in  the 
process  is  to  increase  the  confidence  in  but  not  the  accuracy  of  the  results.  And,  as  discussed  above  and  illustrated 
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with  the  horse  race  example,  additional  variables  are  also  most  likely  to  increase  the  confidence  in  but  not  the 
accuracy  of  the  results.  Judgments  are  inevitably  based  on  a  relatively  few  critical  variables  rather  than  on  the  entire 
spectrum  of  evidence. 

There  are  two  types  of  additional  information  that  are  likely  to  increase  the  efficacy  of  the  planning-decision 
process.  The  first  type  is  information  that  corrects  or  otherwise  changes  the  nature  or  value  of  information  already 
available — such  as  new  information  that  the  insurgency  group  that  had  been  identified  and  located  is  twice  as  large 
as  previously  estimated.  The  second  type  is  information  that  relates  to  the  relative  importance  of  known  variables 
and  how  these  variables  relate  to  each  other.  As  suggested  earlier,  the  mental  models  that  we  use  in  our  planning  and 
decision-making  are  necessarily  simple  due  to  limitations  in  our  information  processing  capabilities.  Thus,  in  reality, 
we  are  limited  to  employing  models  that  can  handle  a  relatively  small  number  of  variables  (between  5  and  9), 
making  it  important  to  select  the  most  critical  variables.  Additional  information,  then,  that  helps  identify  which 
variables  are  most  important  and  how  they  relate  to  each  other  adds  significant  value  to  the  process. 

Within  the  context  of  the  cognitive  challenges  and  considerations  described  above,  cost-value  tradeoffs  must  be 
made  in  support  of  decisions  to  either  act  now  or  collect  and  analyze  additional  information.  If  the  decision  is  to 
seek  additional  information,  further  cost-value  comparisons  will  be  required  in  which  different  types  of  information 
will  require  different  expenditures  in  terms  of  time  and  effort,  and  in  the  potential  risks  of  delays  in  taking  action. 
The  principal  skill  to  be  developed,  then,  will  be  that  of  assessing  alternatives  in  the  face  these  of  planning-decision 
challenges  and  selecting  courses  of  action  that  will  be  most  likely  to  lead  to  desired  results. 

Battle  Command  Tasks  Associated  with  Skill 

The  principal  battle  command  task  for  which  this  skill  is  required  is  that  of  assessing  the  current  situation.  The 
specific  task  elements  include: 

1 .  Identification  of  what  specific  information  is  lacking  or  not  sufficiently  reliable 

2.  Determination  of  whether  or  not  sufficient  information  is  available  to  make  informed  judgments 

3.  Identification  and  verification  of  specific  information  that  is  unusual,  unexpected  or  out  of  the  normal  range 

4.  Assessment  of  the  relative  importance  to  the  plan-decision  objective  of  known  variables  and  how  these 
variables  relate  to  each  other 

5.  Assessment  of  the  tradeoff  between  the  potential  cost  of  developing  additional  information  (time,  people, 
material,  risk,  etc.)  and  the  value  of  the  information 

6.  Determination  of  when  to  stop  seeking  information  when  not  found/developed 

Types  of  Errors 

A  variety  of  errors  are  common  when  executing  this  skill: 

1.  Failure  to  determine  that  needed  information  is  missing 

2.  Failure  to  determine  that  available  information  is  insufficiently  reliable 

3.  Fixation  on  specific  missing  information  to  the  exclusion  of  other  possibilities 

4.  Gathering  additional  types  of  information  that  are  not  likely  to  result  in  increased  judgment  accuracy 

5.  Unwarranted  increases  in  confidence  related  to  the  collection  of  additional,  unneeded  information 

6.  Failure  to  identify  and  verify  questionable  information 

7.  Failure  to  trade  off  the  potential  value  of  information  with  the  cost  of  obtaining  the  information 

8.  Failure  to  balance  time  and  cost  of  developing  additional  information  against  their  possible  effects  on 
the  mission 

9.  Failure  to  terminate  the  search  for  information  on  the  basis  of  value/cost/risk  tradeoffs 

10.  Failure  to  develop  and  apply  criteria  for  terminating  the  search  for  additional  information 

Learning  Objectives  for  this  Skill 

This  training  module  will  focus  on  a  set  of  learning  objectives  that  constitute  aspects  or  components  of  this  skill. 
Upon  successful  completion  of  the  training,  the  student  will  be  able  to: 

1 .  Develop  an  understanding  of  the  two  principal  challenges  related  to  seeking  additional  information — 

1)  knowing  when  one  already  has  the  minimum  information  to  make  an  informed  judgment  and  2) 
developing  an  awareness  of  the  types  of  information  one  is  actually  using  in  the  planning-decision 
process 
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2.  Develop  the  capability  to  identify  what  specific  information  is  missing  or  not  sufficiently  reliable  to 
complete  satisfactorily  the  planning-decision  process 

3.  Develop  sensitivity  to  and  skill  in  the  Identification  of  indicators  of  questionable  information  based  on 
observed  values  of  variables — information  that  is  unusual,  unexpected  or  out  of  the  normal  range — and 
seeking  information  that  will  serve  in  its  verification  or  refutation 

4.  Develop  understanding  and  skill  in  assessing  the  relative  importance  to  the  planning-decision  process 
of  known  variables  and  how  these  variables  relate  to  each  other 

5.  Develop  sensitivity  to  the  costs  (time,  labor,  material,  risks)  of  acquiring  additional  information  and 
skill  in  making  value-cost  tradeoffs  when  considering  the  acquisition  of  additional  information. 

6.  Develop  an  appreciation  of  the  consequences  of  continuing  to  seek  additional  information  when  it  is 
not  necessary  for  determining  an  appropriate  course  of  action,  and  skill  in  developing  and  applying 
criteria  for  when  to  terminate  the  search  for  additional  information. 

Skill  Boundaries 

This  skill  is  related,  to  a  greater  or  lesser  degree,  to  each  of  the  other  critical  thinking  skills  since  each  involves  some 
degree  of  information  gathering  and  analysis.  Clearly,  time  and  effort  spent  to  collect  unneeded  information  will  cut 
into  that  needed  for  the  application  of  these  other  skills  and  also  burden  battle  command  tasks  with  irrelevant 
information.  On  the  other  hand,  not  taking  the  time  and  effort  to  identify  and  collect  needed  information  will  result 
in  inadequate  material  upon  which  to  apply  these  other  skills.  Thus,  inadequacies  in  the  application  of  this  skill  can 
reduce  the  effectiveness  of  framing  the  message,  recognizing  the  gist  in  messages,  developing  explanations, 
generalizing  from  specific  information,  creating  mental  imagery,  addressing  biases,  and  examining  the  perspectives 
of  others. 

Flow  of  Training 

1 .  Start  the  training  with  a  simple  game  that  requires  the  student  to  make  tradeoffs  between  costs  of 
playing  and  potential  payoffs  from  the  actions  taken.  The  objective  is  to  introduce  the  concept  that 
actions  have  associated  costs  and  that  actions  can  have  maximum  payoff  only  if  the  associated  costs 
are  carefully  considered.  This  initial  game  will  not  necessarily  relate  to  military  operations  or 
information  acquisition,  but  will  be  in  a  context  that  should  be  familiar  to  any  student. 

2.  Introduce  the  topic  of  this  training  session — deciding  when  to  seek  information  based  on  its  value  and 
cost.  An  example  of  a  typical  problem  will  be  presented  and  discussed  within  a  realistic  or  actual 
military  context. 

3.  Describe  and  discuss  the  two  principal  cognitive  challenges  related  to  seeking  additional  information — 
1)  knowing  when  one  already  has  the  minimum  information  to  make  an  informed  judgment  and  2) 
developing  an  awareness  of  the  types  of  information  one  is  actually  using  in  the  planning-decision 
process. 

4.  Present  a  series  of  scenarios  in  which  the  student  is  required  to  1)  determine  whether  or  not  sufficient 
information  is  available  to  take  action  and  2)  if  sufficient  information  is  determined  to  not  be  available, 
identify  the  information  that  is  needed. 

5.  Present  a  series  of  scenarios  in  which  the  student  is  required  to  identify  the  five  most  important 
variables  to  be  employed  in  deciding  on  a  course  of  action.  The  variables  are  to  be  selected  from  a  list 
that  accompanies  the  scenario.  The  student  then  indicates  for  which  variables  there  is  sufficient 
information  available  from  the  scenario  and  for  which  variables  additional  information  must  be 
obtained. 

6.  Describe  examples  of  questionable  information  based  on  observed  values  of  variables — information 
that  is  unusual,  unexpected  or  out  of  the  normal  range — and  how  seeking  additional  information  will 
serve  to  verify  or  refute  its  validity. 

7.  Present  a  series  of  scenarios  that  may  or  may  not  have  variables  with  out-of-the-normal  range  values. 
The  task  of  the  student  is  to  identify  these  variables  and,  further,  to  evaluate  the  relative  importance  of 
each  in  terms  of  the  effort  that  should  be  expended  to  verify  or  refute  its  validity.  The  assessment  will 
be  made  employing  a  five -point  rating  scale  graduated  in  terms  of  the  amount  of  expenditure  (person- 
hours)  the  student  would  be  willing  to  make. 

8.  Present  information  and  describe  examples  relative  to  the  costs  (time,  labor,  material,  risks)  of 
acquiring  additional  information  and  skill  in  making  value-cost  tradeoffs  when  considering  the 
acquisition  of  additional  information. 
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9.  Present  information  and  describe  examples  about  the  potential  consequences  of  continuing  to  seek 
additional  information  when  it  is  not  necessary  for  determining  an  appropriate  course  of  action.  Present 
considerations  and  criteria  for  when  to  terminate  the  search  for  additional  information. 

10.  Present  exercises  in  which  value-cost  tradeoffs  are  required  for  seeking  additional  information  relative 
to  deciding  on  alternative  courses  of  action.  For  each,  a  military  scenario  (containing  the  information 
available)  is  provided  along  with  three  alternative  courses  of  action  to  be  decided  upon.  The 
probability  of  success  is  degraded  differentially  for  each  course  of  action  depending  upon  the  time 
delay  in  executing  it — time  is  of  the  essence.  A  look-up  table  provides  the  costs  (person-hours)  for 
acquiring  various  types  of  additional  information  associated  with  different  variables  potentially 
relevant  to  the  action  decision  and  its  contribution  to  the  probability  of  success  of  the  associated  action. 
The  exercise  is  assessed  in  terms  of  the  probability  of  success  of  the  action  ultimately  selected.  (Note 
that  success  does  not  depend  on  knowledge  of  military  courses  of  action  but  on  the  ability  to  make 
trade-offs  with  knowledge  about  values  and  costs  of  additional  information  provided.) 

Training  Exercises 

The  training  will  include  the  following  exercises  briefly  described. 

Training  Exercise  Set  1:  Out-of-the-normal  range  values.  The  task  of  the  student  is  to  identify  out-of-range  variables 
and,  further,  to  evaluate  the  relative  importance  of  each  in  terms  of  the  effort  that  should  be  expended  to  verify  or 
refute  its  validity. 

Training  Exercise  Set  2:  Sufficient  information.  The  student  is  required  to  determine  whether  or  not  sufficient 
information  is  available  to  take  action. 

Training  Exercise  Set  3:  Value-Cost  tradeoffs.  The  student  is  required  to  make  trade-off  decisions  about  alternative 
courses  of  action. 


B-23 


APPENDIX  C:  FORMATIVE  EVALUATION  QUESTIONNAIRES 


Demographic  Questionnaire 


Control  Number 


Rank 


Branch 


Gender 


Estimate  the  number  of 
months  you  have  been  in 
a  combat  zone. 


Post  Secondary 
Education 


Age 


GPA  last  semester 


v  Post  Secondary  Major 


Dominant  Hand 


Internet  Usage- 
Average  per  week 


Which  Internet  Browsers  have  you  used  in  the  past  year? 


Internet  Explorer?  ^ 
Netscape  Navigator?  I* 


Mozflla  Rrefox? 
Other  Browser 


Which  Internet  Browser 
do  you  use  most  often? 


Please  indicate  your  willingness  to 
use  online/web-based  distance 
education  training  courses  in  order 
to  further  your  Army  education? 


What  best  describes  your  work-related 
experience  with  web-based  training 
materials? 


If  you  have  taken  web-based  training 
courses,  please  rate  on  the  5-point 
scale  below  your  experience  with 
these  courses? 


Record:  |  H  ] 


fH>  of  1 
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Example  Usability  Questionnaire 


*•  SWIM  |X 
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